Android Ramblings, Craigslist Adventures, Technical Nonsense

I've been working at Google this summer as an intern, which is part of the reason why there haven't been any updates to any of my things. (This blog, MicDroid, etc) They say Google is a place that engineers disappear into, and are not heard from again, and that seems somewhat true for me this summer. The other part of it is simply that my life has become busy, and I've had to make some sacrifices in what I spend my time on.

If there's one thing that's happened fairly frequently, it's that people are interested in what goes on at Google. Today I'll attempt to talk about some of the things I found interesting. 

Please note: these are my personal views, they are not meant to be official announcements of any sort, nor are they in any way endorsed by Google.

Let's begin with what I've learned this summer at Google.

• Python: My projects this summer were primarily Python based, with a small amount of C++ on the side. I estimate I wrote around 3KLOC worth of Python, and around 500LOC worth of C++. Needless to say it's great to learn another language, and for the Python lessons alone, this summer was worth it. One of the nicer things about learning Python at Google is that "correct" behaviors are enforced by the company style guide as well as code reviews. The best part about Python for my project was that it reduced the logic of the program to data structure manipulation. 
• Scripting ability: Since my project involves managing a fairly large sized fleet of machines, being able to script fluently using the usual bash tools (awk, grep, sed, etc) is important to successfully doing my job. A related lesson to this is to realize when your scripting has gotten out of control. After a certain level of complexity, perhaps it is better to fold the script functionality into a full-blown program.
• Dependencies matter in building large distributed systems: I accidentally triggered a DDoS using the entire fleet of machines because of a set of changes that introduced a dependency on an external service. As it turns out that external service was unable to handle the load of the entire fleet of machines because that was beyond the designed usage of the service. Since then I've been a lot more careful in invoking library functionality that may depend on external services. 

Next let's talk about how things are done at Google.

As expected of the engineering-driven culture at Google, the toolchain for working in the main Google source tree is pretty heavily developed. 

• Google's build system is fast. This blog post series provides a more in-depth look at it, but the short description is that it caches build output across the entirety of the main source tree to avoid having to do unnecessary compilation. Because of this, the large majority of build work done are actually retrieved from cache, leaving the builder very little actual work to do. Rebuilding my summer project typically takes less than 10 seconds.
• Code search is very powerful. This blog post introducing public Google Code Search mentions that public Code Search is based on an internal tool that is very widely used. Internal Code Search is several orders of magnitude more powerful than the public variant, so much so that I typically browse through code in Chrome rather than the terminal.
• Code reviews are required for any code that goes into the main tree (especially for interns!). Google has actually opened up a variant of their code review tool on Google Code. There is also an article and tech talk by Guido van Rossum available as well. I actually liked Google's tool quite a bit, since I basically tracked my tasks and TODOs through it.
• Code style is enforced through code reviews, style guides, and the presubmit process, as mentioned earlier. This leads to code of relatively uniform quality--quite a contrast to many other places I've worked before.
• Git at Google. Google is known for being a Perforce company. It turns out there is quite a bit more to the Perforce system at Google than just that though. Also, the engineers have written a Git interface around Perforce, and it works relatively well. This lets me feature branch as much as I want while allowing me to make bugfix branches when necessary. Very useful.

Infrastructure at Google is actually quite interesting too. Having to deal with a lot of machines also equates to having a good amount of tools to deal with them as well.

• Google's VM management system is called Ganeti. It's been open-sourced, and the project page details that it's built around Xen/KVM. Ganeti is a layer above Google's vast datacenter resources. There's a fairly well developed set of tools around making Ganeti programmable as well.
• Building and deploying to our machines is done through a combination of a constant integration system that produces debs, APT, Puppet, and Slack.
• A good sysadmin needs to not only know the system inside and out, but needs to be able to script and program their tools as well. This is something I really need to work on.

In addition to being known for engineering prowess, Google is also known for being an interesting place to work.

• It's typically said the company is opaque to those on the outside, but transparent to those on the inside. Engineers are given access to almost everything, including source, documentation, information (including non-engineering related info), etc about nearly all Google projects. Additionally, there's no particular embargo on talking to engineers working on other projects, so it's not uncommon at all to hear a lot of details about other projects over lunch. There's just a general openness regarding information inside the company.
• There's just as much complaining about Google products inside the company as there is outside. Especially with the recent launch of Google+ and the inconsistent real name policy that accompanied it, there has been just as much debate inside the company as there has been outside. So to those of you who have fallen victim to the real name policy, we feel you.
• Along with the openness of information, internal public discourse about what the company is doing has many avenues, from mailing lists to groups in the company, to all company meetings. It's great to be a part of the larger discussion concerning what the company is doing.
• Google is a very strong proponent of dogfooding. Google employees can (and do) test early builds of almost any Google product before general release. Along with this comes the discussion of future features and products, of which there is quite a bit going on.
• At least for the teams I was working with it felt like Google was very bottom-up, where we work on a small area of the team mission each day, rather than take direction on a specific product or feature. I'm not sure if this is due to my team not being a strict product team, but it sure does feel like each day you get to choose to devote your time to something that furthers the team mission. Personal ownership of what you're working on is also pretty high, at least in the team that I work with. It's definitely not on the same level as a startup, but it's quite a bit nicer than some of the other places I've worked at. Word across the internet says things are changing across Google in this regard though, but it didn't affect the teams I was in touch with.
• There is a HUGE amount of information inside Google. This information is about nearly everything Google related, from product information, code documentation, tutorials, internal tools, what's for lunch, parking, traveling, personal projects, etc. Not all of it may be up to date, but the information is definitely there. One could waste days just learning about Google history, how Google works, etc by reading documentation and sites inside. 
• There is a slide in one of the Google Mountain View buildings. A slide. I couldn't find it. Maybe next time.

I'm definitely going to miss working at Google (and not just for the free food!).

• I am going to miss the (relatively) consistent code style. Having worked at several places where there was no real style guide, being able to actually READ other people's code was quite pleasant.
• The tools (those mentioned above, and some that weren't) make life quite a bit easier, especially the code search. I am seriously going to miss how useful the code search system was. In fact I'm already thinking about a possible system that hooks into Git repositories...
• Working on a large distributed system is something I haven't done before (the systems at Supercomputer Center were quite a bit smaller). It's got it's own unique issues, but overall it is quite fun to be able to work on.
• It's easy to find information at Google (who would have guessed?). Even if what you find is not current, at least it gives you a starting point to get more information. Additionally, there is a LOT of really fascinating information lying around internally. I just wish I had more time to ingest more of it.
• Free food. (naturally)

There are a few things that I did find irritating while working at Google though. Here are some.

• Some common usages of infrastructure require approval. This takes time away from useful implementation time and slows down the overall pace of work. I can understand why they require approval, but sometimes it seems rather arbitrary. 
• Not all the information is up to date. It's very easy to find old documentation, or deprecated systems. Sometimes there isn't even new documentation yet. This can also make it difficult to find the RIGHT information. Usually the solution here is to ask, but it nothing breaks up your workflow like finding out you were using a deprecated or undesired feature.
• Not Invented Here syndrome is supposedly one of the big problems Google faces, however I didn't actually run into it too much (or maybe I don't know of enough outside systems to know I'm running into it). I do think that the existing code/build/repository model makes it difficult to integrate 3rd party code however. 

Overall, Google has been great to me this summer, and I really think I would enjoy working there in the future (provided this post doesn't disqualify me, oops). 

I've heard T-Pain himself mentioned that the famous I Am T-Pain app will be coming to Android, and that there will be much rejoicing. What does this mean for MicDroid? Well, despite the fact that I definitely don't have the backing of a company to equalize on features, I don't plan on closing up shop. It will however let me work at a more leisurely pace (as if the pace I work at isn't leisurely enough already), because there are a few things which are interesting enough to me to keep me working on this project.

Those of you out there who actually keep up to date with my commits on github will probably have noticed that MicDroid is in fact capable of background music now, albeit with some pretty serious caveats. First and foremost is the fact that in order to set a track as background music, it is required that the track be in WAVE format, and not MP3, AAC, or any of the other less popular audio formats. This is primarily because Android does not expose any sort of audio decoder (or encoder) API to developers, despite the fact that most phones have hardware capability to do so. The current workaround is to use Lame4Android to decode MP3 to WAVE, then set that WAVE as backgrond music. Ideally I'd like to add liblame to MicDroid and just decode MP3s on the fly, but the MP3 codec does in fact have licensing issues, and I'd prefer not to deal with that for now, at least not until other more interesting problems have been solved.

The other two remaining issues that I'd like to look at are echo cancellation and audio resampling. Both of these are necessary since phones may be recording at an arbitrary sample rate, while background music may be playing at a different sample rate. Since MicDroid uses a simplistic Output = Mic / 2 + Music / 2 sort of formula to mix, this means the PCM data input to the mixer needs to be sampled at the same rate. Currently I'm looking to tap CCRMA's resample library to do this, which means reading and understanding the code. Sadly my background is in software, so I don't have the DSP knowledge to make it a trivial task, so this may take longer than I hope. Secondly, the echo cancellation, which will make it far easier to deal with live correction (hopefully), is something else I'd love to add in. Personally I'm actually not too sure there's a lot that can be done, due to the lack of any low latency audio API in Android, but I think it's a direction worth looking into. The best sources for this seem to lie in VOIP software and solutions, and I'm looking into sourcing some code from Oslec and csipsimple. Again, DSP is not my strong suit, so it remains to be seen how much I can do with the code. 

So in summary, there are still plenty of things left to build, and plenty of directions to expand. I'm thinking that once I get basic audio mixing capabilities functioning using resampling I will release an update with instrumental support as an experimental feature. Additionally, it is my plan to do a write up about the various audio processing that MicDroid does as well.

Until then though, I have plenty of coding to do.

I've put an updated version of Lame4Android out on the Android Market today, and while I'm glad to say it now supports decoding WAV->MP3, it does still have one small glitch, where feeding in files created using Lame4Android will not encode properly. I have a feeling that this is due to the first few bytes of the file not being skipped correctly, which causes the encoder to crap out immediately. I will have a fix for this soon. 

Now, on to what I wanted to talk about. Any programmer over the span of their career will have written an API of some shape or form. It may not be good, or widely used, but they'll have done it. This is one of those parts of programming you wish they taught everyone else in school, along with variable naming and spacing, in that you'll wish everyone wrote their APIs like you would. It's also one of those things where everyone can tell when there is a good API, but cannot necessarily write one themselves. 

I've had plenty of experience (attempting) to write APIs, but the first time I truly had to worry about what I was writing was when I worked at Deluxe. At the time I was given the task of re-writing the framework on which all of the Universal Blu-Ray releases were written on, since the previous version was getting a bit long in the tooth, and did not support some of the newer base framework functionality. Additionally it was full of strange quirks and unnecessary wrapper functions. Now here I think is the crux of writing a good API: exposing the right amount of magic to the client program. What I mean by this is that the amount of magic the API does behind the scenes to provide the functionality available underneath must not be too little, nor too much. Too little magic in the API and you're left with a nothing more than a thin wrapper around the underlying functionality. An example of this would have been the Universal Blu-Ray framework I was talking about earlier. In quite a few places the old framework would provide wrapper functions to switch audio tracks (think switching from French audio to German) that was nothing more than a wrapper around the underlying base framework's switch audio track functions. On the other hand, too much magic in the API leaves you with a very inflexible set of exposed functionality. An example I'm thinking of would be the Android MediaRecord API. While the designers expose pretty much all the functionality a developer would need to record audio, it just isn't very configurable. You can only record in a certain format, and then only directly to a file, not to a buffer, or to the speaker, or anything else. Nor can you (not until Gingerbread at least) apply effects to the recorded stream. In order to do any of the above, you as a developer must use AudioRecord, which only outputs to a buffer that the developer must manipulate. Too much work is done by the underlying layers, all of it magical to the client programs.

Now to get back to the point of everything though. Today I'd like to say the liblame API is finished (or as finished as a hobby project ever gets), and I'd like to say I did a good job in exposing enough magic to anyone who would like to use it. Please let me know if you do, I'd love to hear from you.

I've recently been bitten (again) by Samsung Galaxy S AudioRecord bugs, and after having to ignore it to focus on classwork for the past few weeks, it's time to get back into it. The last MicDroid update featured improved error handling due to proper (sort of) use of exceptions instead of an Android Handler to route all exceptions to error handling code. Unfortunately this broke Galaxy S support due to what I believe is described in these posts. It appears the Galaxy S phone just locks up the AudioRecord if you try AudioRecord.startRecording() while it is initialized improperly. This behavior certainly seems consistent with the (mass of) bug reports I've been getting from users. I think I'm going to have to try some of the dirty hacks mentioned above to fix it.

Another project which appears not to have had too many issues with the Galaxy S series is the rather famous Sipdroid. They also have AudioRecord code which supposedly works on the Galaxy S series. They use an interesting method of delaying until the next frame is read before reading from AudioRecord again. I'm fully intending to try this method out also, since if it works, it means I can potentially simplify a lot of the complex recording code current around. Also, it could potentially mean fixing the horrible buffer size hacks I'm using right now and finally allow the Galaxy S series to take advantage of live recording.

Should I get it to actually work, I definitely plan on releasing details.

Finally, the last, best hope for the Galaxy S series is the upcoming release of an official 2.2 ROM. There have been rumors that the 2.2 ROM fixes a lot of the audio bugs that plague the 2.1 Galaxy S ROM, and should an update that fixes these fully roll out, all of these horrible Galaxy S problems should be solved. Obviously this is the best solution for everyone :)

Something else I've been throwing around in my head has been the fact that the Galaxy S has a relatively slow internal SD card. I'm wondering if this has any effect on recording, especially if AudioRecord.read() reads audio at a rate faster than the phone can write. 

It's problems like these that really have me looking forward to Gingerbread, as it's OpenSL and effects pipeline support can potentially go a LONG way towards resolving basic audio functionality issues like these. Now if only we could get Google to release the source code and Samsung to follow standards and release fully baked hardware...

MicDroid 0.40 has been released!

Release highlights include many force close errors fixed, including some more weird async task related rotation issues. Also there have been ads added to the recording library. AdMob ads can be removed via a setting in the options however, they're purely optional. Code for instrumental support has been added, however it is not enabled, since it only supports wave files currently, and that isn't much use for most users. Future work primarily involves getting LAME built as a library for proper mp3 support. Hopefully this release will be relatively trouble-free, and have fixed most of the outstanding issues from 0.39. Thanks for your support!