Emil Lerch

wttr.in

Fri, 09 Jan 2026 14:43:51 -0800

wttr.in

This is another heavy AI-assisted coding post…sorry about that. But, while AI is featured in the story, it’s not the point of the story. For a long time, I have been a fan of wttr.in. I’m posting the github link here instead of the production site, mostly because it takes 5-10 seconds for me recently (I don’t remember this before?). The concept is really cool (who doesn’t like weather you can use from your terminal). I cloned it with the intention of self-hosting it, and…learned a lot.

Installing

So, the intro of the readme says in part:

Originally started as a small project, a wrapper for wego, intended to demonstrate the power of the console-oriented services

That shows. Here’s the (AI-generated) request flow from wttr.in. It actually get’s a bit more involved than even this:

Client Request
    ↓
Go Proxy (port 8082) - LRU cache + prefetch
    ↓ (cache miss)
Python Backend (port 8002) - Flask/gevent
    ↓
Location Resolution (GeoIP/IP2Location/IPInfo)
    ↓
Weather API (met.no or WorldWeatherOnline)
    ↓
Format & Render (ANSI/HTML/PNG/JSON/Prometheus)
    ↓
Response (cached with TTL 1000-2000s)

So, we have a go program, with it’s own cache. Then a python backend, which, if memory serves, also has a cache. However, depending on the format you’re looking for, it’s either a) rendering itself, or b) kind of sort of skips all this and just calls a fork of “wego”, which, as mentioned in the readme, this was originally the plan as a wrapper. All that other stuff kind of organically grew over time. Self hosting ultimately wasn’t terrible, but it wasn’t pleasant either. Several API keys, and paid services, etc. Also, two separate sets of caching layers, each in their own directory, made creating a dockerfile…well, weird.

A hard blocker

Once installed, I ran into a firm block for me. The default (only at the time I think?) service was a paid weather service. That makes no sense for me as a single user. So, I put my ~money~ time where my mouth is, and picked up the pen to implement met.no as a weather source. They don’t even require an API key, just “be nice”, which is awesome. I didn’t realize about this wego thing, so while met.no worked fine for my purposes, the full rendering was always different from the custom rendering that I was using, and I never really bothered to track it down. In the last 3 weeks, I’ve learned that’s because wego is a very different pipeline, and uses its own web service configuration!

5 years go by

This has all been bugging me for…well, 5 years. On my long term todo list was to help clean this organically grown architecture and simplify it, make it self-hosted friendly, etc. So I find myself with some spare time over Christmas break, and I have a codebase and an AI to help do a lot of typing. So I ask the AI to build a plan based on a new, simplified architecture, and we work together to come up suitable. It’s original estimate was 10 weeks for a full rewrite. I have a few missing features (i18n, PNG support, Moon endpoint…but moon phase is supported, and wttr.in compatible json, as my json is a met.no passthrough).

Starting out

I asked the AI for a plan. I wanted this built in zig (of course). I wanted a vastly simplified architecture. At the end of the day, a single caching layer (vs Go+python each with their own) in a single directory. Everything will be self-contained, with the goal as a single static binary. Spoiler: I’ve achieved this, even with the first pass of png support. I wanted a coherent organized code base (still some rough edges there imo), and critically, I wanted unit tests everywhere. The AI made up a goal of 80% test coverage, and my coverage report currently says I’m at 85%. I had it document the current architecture and the target architecture, so I could argue and eventually just edit the target architecture for future use. Then, I let it rip.

First pass (6hrs)

I got something functional, but there was a lot of cleanup to be done. In 6 hours of work, mostly me arguing with the AI, I had a full working (but admittedly buggy, and with structural deficiencies) implementation of most of the main features. This was literally “close enough for government work”, that I felt able to delete all the python/go in the directory to focus on the zig implementation. Any other discrepancies, I’d compare to the live https://wttr.in/ site or pull up the github repository. And I have to say, the live site…I’m not sure if it’s a result of the architecture, zig, or simply that I’m only serving 1 req/sec, but the performance difference between this code and the live site really made me reluctant to check out the live site unless absolutely necessary.

Refining (15hrs)

My process for AI use on greenfield is often like this. Spend 5-10 minutes getting the AI and I on the same page for what needs to be done. Then let it write whatever it wants, just make it work. Then start in on the more detailed work, where I still use AI (more later), but I start getting a lot more hands-on. Typically, I think of this the same way I imagine a carpenter would. Chainsaws, then more specific tools, followed by sanding, staining, etc. This initial pass at this was about 15 hours. Life then intervened, and I took a 2 week break.

Finishing up (1 week)

After the break, I spent a few hours every day for about a week, and wrapped it up. At this point, AI was still heavily assisting me, but I was no longer checking in any code that I didn’t code review and hand-edit to something I would have authored directly. I literally said to myself on more than one occasion, “this works, but it doesn’t look like code I would write”, and I’d change it. Sometimes I’d get into heated arguments with the AI, to the point where once my wife came downstairs and asked why I was typing SO DAMN LOUDLY. I had to tell her that was me yelling at the AI.

Things that were super helpful:

Large refactorings: Once all the unit tests were in, the AI could cut through these like butter, but I did have to make sure it was very clear when it could and could not adjust test expectations. An easy way to get tests to pass is to just adjust the expectations!
Suggestions on where to start: I don’t know anything about calculating moon phases or dawn/dusk/sunrise/sunset. The AI gave me a good overview, then pointed me to C code in both cases. And it was able to quickly integrate them into the project for me.
Giving me multiple suggestions. Not sure of a design approach, or a name? Ask the AI for 3. Ask it to explain the tradeoffs of each, and what its suggestion is. I took the suggestion only maybe half the time, but the process helped me think through everything.
Doing a lot of work quickly. Even if I know it’s going to be terrible, I’d rather have it write 20 unit tests, and then go fix the std.testing.expect(response != null) than to have to write all the boilerplate and setup stuff 20 times myself. Keep in mind here, that part of writing the tests might be (sometimes large) refactoring to components to allow for things like test mocks, or to separate things doing IO from pure functions. I knew that whenever the AI finished, the build and the tests would succeed, and I would just play quality control.

Where I struggled:

Visual stuff: It was easy to get it to get something visible, but getting it right was a whole different matter. I had a day-long argument about whether monospaced ascii was aligned when it include the lighting bolt (⚡) character. Even when I provided the string in a test expectations, it just refused. Eventually, I got it to tell me why, and it’s not for the reason you might expect (multi-byte unicode characters). No, it boiled down to “crappy terminals don’t know how to display this character, so it should take up two cells. Therefore, the right thing to do is to remove a padding space”. ok, wow… so after telling it that I don’t care about buggy terminals, that’s on them, it was able to align things correctly. Once all the proper mocking code was in place and I had (hand coded) the proper unit tests, this problem went away.
Adding new zig dependencies: This is a nit, but it consistently screws up a simple zig fetch --save. After this bit me twice (noticed it in CI), I learned I could do the command myself much more quickly than having it botched and either not catching it or having to argue about it.
Style: The AI really wanted to make subdirectories. It really liked to do things like have a geoip.zig file with a GeoIp struct inside it (the zig way is to call it GeoIp.zig, and the file becomes the struct, but the AI struggles with that concept. The AI really wants to put braces around everything.
Code duplication: I’m not a hard core DRY person, but the AI thinks nothing of writing the same code 15 times in 15 different places. This can introduce a lot of specific bugs and sometimes lead to dead code. I’m still thinking of the three different caches in the system, which each have their own implementation, and wondering if it’s appropriate. I think it is, as each have unique aspects to the data, but I’m not 100%.
Terrible test expectations: This was just consistently terrible, and eventually my expectations were that for unit tests I would just need the AI to blast out some stuff, then I would do the final work on asserting the right expectations myself.
Ignoring instructions: Sometimes the AI would simply ignore me, or not do the research (file reading) I would have expected it to. This was the most frustrating thing, and I think it boils down to “my context was getting too long”. I learned to restart my session, and keep sessions focused on one task. If I asked it to refactor something, add in two different pieces of new functionality, I would be in for trouble.
Systems work: This was a user-facing application, and there is a lot of prior art for application-level stuff, so I had a pretty good experience. But whenever I use AI for systems level things (e.g. unikernels), it goes bad in a hurry. I don’t even try much there.

Conclusions and future work

This was a good experience, and I’m really excited to do almost a full rewrite in about 2 weeks. I have a docker image for this, that is from the scratch image with two files - the binary and TLS certificates. Adding PNG support, the biggest remaining feature, looks to retain these properties. My prompt for the png support literally started with “What would you suggest for png generation?”, and in 30 minutes I have something working (to be clear, the output is terrible). I suspect I’ll learn a lot about PNGs in the next few days, but I am certain I’ll finish this feature a lot faster than I would have before AI. I will also, of course, make sure the owner of wttr.in knows about this in case there’s any interest in…a full rewrite of the code into a language they’ve likely never used?

Oh…and the repo is at https://git.lerch.org/lobo/wttr

You can try it out at https://wttr.lerch.org/

Building an offline, limited version of Alexa

Thu, 16 Oct 2025 12:50:20 -0700

Building an offline, limited version of Alexa

(or how Belkin and Alexa conspired to nerd snipe me)

It all started when Belkin decided to end wemo support. I am leery of things that require cloud support, whether they are games, or devices. I get that business changes and these require ongoing investment from the company’s perspective, but my interests are unique and different from company interests, and I’m not into obsolescence, especially of physical devices, that work just fine.

There is a disturbing trend of requiring more and more Internet access for things that really don’t need to, and I actively push against this in most cases. There is no reason that I should need my Internet connection to be up and operable to turn on/off my lights! The original Belkin Wemo was exciting, because it could operate locally and Amazon Alexa, though it is cloud-based, will connect to these devices without round tripping to the cloud. If my Internet connection goes out, I still have a physical light switch, and I could even use my phone with Home Assistant. Under normal operation, I don’t want Home Assistant in the mix, because it’s another point of failure, and the Home Assistant/Alexa integration utilizes Home Assistant cloud, so now I need an Internet connection and 2 online services to be operable…for my lights?!

Anyway, for a long time I was in a pretty happy state. I had local control of many lights in my house, Alexa could control them, home assistant could monitor and control them, and all was good. My main problem was getting new Wemo devices, because the later generations of Wemo dropped the local support altogether, and at least once I ended up paying 3x the price of the Internet-only device to get an older device shipped to me. Later, I discovered Tasmota, which can pretend it is a Belkin Wemo device, and CloudFree, an online shop that ships pre-configured Tasmota-based devices, and about half my light switches (and holiday light setups) use devices from them.

Conspiracy Theories Start Here!

With this background, the news of Belkin’s removal of Wemo support drops, and… nothing changes. I have devices that work, I don’t need the cloud, and unless Amazon actively moves to remove the local Wemo control (possible, but unlikely), my life literally does not change. Except…within two weeks of that announcement, I have not one, but two Belkin Wemo light switches fail. Coincidence? I think not! (just kidding, I’m sure it’s a coincidence…but wow is that timing suspicious).

Time to migrate

So…I have two switches to replace. I happily go to CloudFree (actually…I got replacements from Amazon). I did some research and apparently products from Martin Gerry have Tasmota pre-flashed, so I got two of them. Mostly this had to do with…well, I didn’t have a working light switch, so I wanted it kind of…now.

I installed the light switches, gave them the same name, enabled MQTT (zero-configuration for Home Assistant), Belkin Wemo support, and was up and running. With everything except Alexa. Alexa saw the devices but considered them offline.

Investigations

Home assistant was fine. Operating the light switch was fine. Alexa saw them…fine. But absolutely, and still to this day, refuse to operate them. After an inordinate amount of time, I’ve learned why. While the IP address was the same, the name was the same, everything was the same, there was one difference. In newer Tasmota firmware, the Belkin support is on the same port as the web interface. And Alexa stubbornly only likes to connect on (its original port? or does it have to be 49152-49154?). So, either I move the literal switch web interface to that port, or Alexa will refuse to operate it. At this point, I’m fully in both nerd-snipe and stubborn mode, and I’ll be damned that Alexa will tell me what I can or should do with my devices.

A bit of technical background here…Alexa has always been a bit finicky with home devices. When you ask Alexa to discover devices, it sends out a broadcast message for UPnP discovery and listens for responses. In those responses, the Wemos provide an XML response with a lot of details, including the URL with the port they operate on. Because this is a broadcast message, with a larger wifi network and a lot of devices, not all the device data is returned all the time, and that’s… a bit of a mess for Alexa. However, as far as I can tell, either Alexa a) ignores at least the port portion of the url returned, or b) forever remembers the port in your account settings somewhere, even after deleting the device and re-adding it. Neither of which are ok to me.

Having some time, I decided to solve the problem with the nuclear option…

Replacing Alexa for device control

This was a good personal/work project, as I am always looking for projects to exercise generative AI for software development to help keep grounded in truth on the state of the art, since I advise a lot people on this topic.

So, I pulled out my coding assistant du jour, and I worked with the agent to create a specification. I planned to handle this in 2 parts:

Develop a speech to text service that would listen to the microphone and get words
Create a program to interpret the words, and when a device command was provided, well, it would command the device

I don’t know anything about either audio processing, speech to text, or natural language processing for parts of speech, but luckily I had the help of various libraries and the collective knowledge of humankind wrapped in an LLM.

Step 1: Speech to Text

Most modern suggestions will suggest the use of Whisper for speech to text, but my needs are simple, English only common words, and as such does not require that much complexity (or computing power). But speech to text is a problem that’s been worked on and solutions existed as far back as 1997 and probably earlier. Ultimately, I found Vosk, which has several models suitable for English, and used the 40M small model. Because I like Zig, I instructed the agent to use that programming language and pull in the necessary libraries, which included ALSA and Vosk.

Any time I run into incorporation of shared libraries, I struggle using AI assistance, and this experience was similar. Ultimately I was able to convince the AI to create a zig build of alsa-lib, but ultimately failed to do something similar for Vosk, so ended up depending on the pre-built binaries, both with Kiro and with the Amazon Q Developer CLI, which at the time was using Claude Sonnet 4. Since then Sonnet 4.5 has come out, and is a significant step up, but I’m still pessimistic. The Vosk build is somewhat daunting with many dependencies, most of which are pinned versions of forks of other libraries, and natively uses CMake.

I spent a fair amount of time cleaning up, but the AI definitely helped out before I understood audio processing (I have learned a lot but am still a bit shaky on the topic). At the end of the day, my biggest disappointment is that I could not optimize this to work on the Raspberry Pi 3B+. I’m confident a Pi 4 would work, but I only had a compute module on hand, and the microphone wouldn’t work with it due to USB support issues (and I have a weird carrier board).

Because of this I was faced with purchasing a Raspberry Pi, but ultimately this fanless N150 is about the same cost, maybe slightly more, after fully kitting out the Pi. I suspect CPU performance is only slightly more (well done, Pi!), but the I/O performance will be significantly higher on the N150. At the end of the day, this part of the process consumes 2-5% CPU and about 450MB RSS, so a Pi 4 should be perfectly fine keeping up. The code lives here, and the program simply executes a child process to hand off the text, so it’s pretty modular: https://git.lerch.org/lobo/stt. I’ll also note that there are non-English models for Vosk, so while it is not completely generic, it is far more useful than my own “English only” requirement. The code is ok…I did a fair amount of cleanup work from the AI, but then used AI again later in various forms, which added functionality and some more stuff to clean up. In general, my pattern for these things is to progressively take on more and more of the coding from the AI. Here, that happened, but I let AI take on some larger tasks toward the end as I learned more about audio processing through the issues I had with the Pi 3.

Step 2: Understanding the text

Step 1 might have had its complexities, but understanding the text is quite a bit more complicated. Parsing the sentence was not actually the difficulty here, the problem turned out to be learning through experience how words would be misheard by the Vosk recognizer. A phrase like “turn off the bedroom like” doesn’t make any sense and will parse, but parse badly. So you need to look for these common incorrect words and replace them, which itself can be a bit tricky. The amount of tuning for both the replacement algorithm and the words themselves took maybe 15% of the overall project.

Again, AI helped with a lot of the stuff I didn’t want to do. Zig’s current TLS implementation is not yet complete, for instance, so I needed:

A custom download step using curl
A custom program to tweak some of the downloaded code
Get all this integrated into the zig build (build.zig)

At AWS we throw around the phrase “undifferentiated heavy lifting” a lot. The download step is purely throw away code (once zig TLS is complete), and only exercised during a build. It’s necessary, but my idea of a quality bar on this particular code is pretty much in the “does it work consistently? yeah? Then I couldn’t care less what it looks like” space. I ended up needing this on both of these projects so I did clean it up a little bit, but otherwise, I just let the AI “make it work”. The “sed lite” that I had the AI write was kind of the same idea. It was a one-off as I’d like to pretend this might actually build on a Windows host (which has “curl” available enough to do the download), but Windows to my knowledge does not have any kind of sed, real or pretend.

Link was my answer to sentence “part of speech” processing, and the code was very…researchy, but it worked, and it was damn easy to compile. Nothing like a Makefile that compiles a bunch of c files to objects, then link the objects into a library to make your day. Understanding the data structures and nomenclature they use…that was a different matter, and we were in way to niche of a space for the AI to help. But it was able to whip up a console-based sentence parsing diagram on the first try, which helped the debugging process a ton. Again, this was just for debugging…so pretty much “does it work?” was my quality bar. It’s a big chunk of code I’ve not really looked at. On the other hand, I completely wrote the word removal/replacement function as I experimented with different versions of the phrases I’d like it to process. AFTER writing it, the AI got hold of it, refactored it to my direction, and I used a copious amount of git add to provide myself checkpoints if it trashed the code.

AI also did a pretty good job with sending of the commands to the Wemo device, and I love AI for “here’s some random protocol I may (or may not) know, but I know you do. Just go write a function to use that protocol”. My experience is that it usually gets it right, or really close, on the first try. And since this didn’t need to be hooked up to a live microphone, it was pretty easy to add a bunch of unit tests for various scenarios. Zig’s builtin.is_test even let me test that commands would be sent properly without actually sending commands out on the network. Interestingly, AI was unable to reason through that particular piece, but was able to run with the idea once I coded it.

Shipping

The process for getting this out the door was pretty mundane. Compile the code for a baseline x86 cpu in release mode, copy both the speech to text and parts of speech (pos) binaries out, create a service user on the device and a systemd unit. Nothing fancy. I purchased a cheap USB microphone and tested it, learning a lot more about audio processing in the meantime (e.g. some mics report 1 channel, some report 2, and they have specific frequencies they can work at, so you might need to down sample). I did a fair amount of tuning while my wife was on a business trip, because this needs to be fairly reliable before she starts using it. There were some problematic deadlocks, failure handling, etc, but one major change I did was to switch from continuous speech detection to only process audio once noise is detected. This was a major learning for me (audio processing noob), and something that came up in an AI chat. It was nice to have AI make this kind of major change to the codebase, and it continues to be a pleasant AI experience. Often, significant refactorings…well, the effort doesn’t seem worth it, or we’re too lazy or busy with other things, but I’ve found it’s much more likely for me to be willing to move, rename, and redesign (beyond what refactoring tools can do) a load bearing central component of code with 1000 references if I can tell AI to go do it while I work on something else, knowing it won’t stop until the build (and its tests) successfully complete. I’m in cleanup mode for sure, but it will have done 90% of that work for me. Ultimately, a week or so in, I was left with a couple things:

Continued need for tuning all the ways it can mis-hear speech (this might be where Whisper does better?)
I think there’s a slow memory leak, which I ignored in favor of letting systemd just restart once/day

I haven’t touched the thing in 3 months…it just…works. It’s not generic like Alexa, and the speech still isn’t quite as robust, but it works!

How I work

Thu, 26 Jun 2025 16:58:11 -0700

How I work

I had someone ask me a while ago “how do you work, exactly”, because my setup was strange and new to them. It’s actually an interesting question, and it’s evolved over more time than I’d care to admit. Because it evolves, this post is a snapshot in time, representing my setup as of mid 2025. I’ll start with an overview, before getting into the nitty gritty details.

Overview

At work, I use a Windows virtual machine, using technology from Amazon Workspaces. This is a special Amazon version we call “corporate workspaces”, and gives me the freedom to run whatever I need to as a client. I originally embraced it due to conflicts between Docker and the Amazon VPN, but I found it useful to use a desktop at home, and laptop when traveling.

At home, I have a primary desktop I use, and a couple secondary desktops that are in use by other members of my family. One of these runs Windows for my son to play some games, and the others all run Linux. I use the Windows desktop for games sometimes, though I’m starting to play games on Linux. My work laptop is a Mac that I use primarily for customer demos, testing, and I bring it with me occasionally traveling for the same reasons.

The rest of this post will primarily focus on Linux, although most of what I write will also apply to Mac. Theoretically my setup will work with other operating systems such as FreeBSD, but I haven’t tested this, and my bootstrap script (more later), would definitely need changes. Philosophically, I try to stick to as much open source tooling as possible, and I value the ability to quickly get to a familiar environment, and update to my latest set of tools, with no surprises. Sticking to a combination of scripting and open source tools help immensely. However, for work, this isn’t possible and as a result, I don’t even try. Avoiding Outlook for instance is possible at Amazon, but it is too much work, and I couldn’t realistically function without Word and PowerPoint in my role. My corporate workspace serves as a good partition, however, so I lean on that. To have maximum portability/flexibility, I want to be able to have the same set of tools everywhere. Sometimes that means over ssh, so over time I’ve become focused on CLI/TUI workflows.

With this in mind, my general approach to using software is the following hierarchy, rated in order, so I will look for software starting at the top of this list, and traverse down, stopping as soon as a suitable option is available:

Open source CLI
Open source TUI
Open source GUI
Proprietary CLI
Proprietary TUI
Proprietary GUI

Because proprietary software is…proprietary, I trust it less, especially in recent years as telemetry has gotten commonplace, and I just don’t feel like people should be recording my every move. Also, I can’t tell what else the software might be doing, and while Android and iOS have deep permission structures with user consent, no such system is built into desktop operating systems. So, with very few exceptions, I sandbox proprietary software in one of the following ways:

At this time of this writing, I believe the only exception to the Flatpak/ Distrobox sandboxing is NVidia GPU drivers. And the only distrobox usage at this time is for Amazon Workspaces. However…I do have one piece of Amazon software (an unreleased product), installed through more native means. This is not open source, but as an employee I actually do have access to the source code, so it’s mostly aligned with my philosophy.

For open source software, I will install CLI/TUI tools through nix, and GUI tools through the following means, usually in this order:

nix
AppImage
Flatpak

I do not generally script most of my base OS installations, but I have scripted out base utilities to manage the rest of my setup. This includes my home directory and configuration, including nix configuration (but not the nix utility install). Using a random linux machine with internet access, I can be productive with most of my tools after downloading and running my bootstrap script. To get all the CLI/TUI tools I’m used to using, I can install nix and type nix-configure, and any updates in my configuration are handled by mr up && nix-configure. AppImages are installed via nix, and flatpak…I just have a text file with common stuff. Workspaces is a manual process for the moment.

Nix focuses on determinism, so by centering as much on nix as possible, I minimize the amount of surprises in my life. Moving from machine to machine, as long as my configuration repositories are at the same commit, all my tools will be at the same version, and I’ll have all the same bugs/features in my tools everywhere. For operating systems, I stick to Debian where possible, and install as little on top of the base system as possible. The last time I rebuilt, I captured a text file with the apt packages I install - it’s about 26 items to handle local hardware and a base X Windows configuration (I have not switched to Wayland). However, my laptop runs PopOS! which is much more maximalist. I don’t really care, because nix-installed tooling takes precedence over whatever was installed out of the box.

How does this work? Part 1: Bootstrap and configuration

Starting from the beginning, my bootstrap script, which hasn’t changed since 2021, installs some important base utilities:

mr
vcsh, pinned to a prior commit
git
Other dependencies, if necessary

The other dependencies include curl (to download mr/vcsh), and perl (which mr uses). vcsh, at the pinned commit, is a set of shell scripts. Beyond that commit it has changed architecture. These two tools are critical to all my configuration, and roughly follows the pattern outlined in the advanced configuration of this blog. tl;dr, vcsh allows me to manage multiple git repositories in the same directory (here, $HOME), and mr allows me to quickly manage multiple repositories in that directory. While I end up with a lot of repositories (mr manages 39 repos at the time of writing), I like the fact that I have a configuration repository for each tool I use, rather than a single dotfiles repo as many recent tools guide you to using. I’m likely swimming upstream a bit here, but I am mostly happy with my setup. My repos on github all have the prefix vcsh_, which was a mistake, and GitHub has special treatment for a dotfiles repo, which is not incompatible with my setup, but I have not configured this as I am not a huge codespaces user. I actually like the concept of codespaces, but I always have my tools at my disposal. ;)

So…what are in these 39 repos? There are a few raw repos outside vcsh for things like liquidprompt and zsh autosuggestions, but primarily these are configuration files. Right now, I have 32 configuration repositories:

Xmodmap
Xresources
alacritty
aliases
asoundrc
awsaliases
bash_profile
bash_prompt
bashrc
commonrc
desktop
dunst
exports
fehbg
functions
ghostty
gitconfig
gtkrc-2.0
i3config
i3status
liquidpromptrc
mdlrc
mlterm
mr
muttrc
nix
tmux.conf
vim
xinitrc
xprofile
zellij
zshrc

Many of these I haven’t touched for years. Several of them are for programs I don’t currently use (e.g. Alacritty). But they’re all small and self-contained, and if I cared, I could configure this out of mr easily enough. My bootstrap clones the myrepos repo, then myrepos can do the heavy lifting from there.

How does this work? Part 2: Working with shells

mr manages my shell configuration. I prefer zsh, though I don’t actually use many of its special features. I do love the history sharing and a few other niceties. However, zsh isn’t usually the default when I’m encountering a new machine, and I want to get from 0->productive as quickly as possible, so it’s important to me that bash work just fine. Today, bash and zsh look and feel nearly identical to me.

The way this works is through the use of a common file that works for both bash and zsh. Both .bashrc and .zshrc source this common .commonrc file, and only commands that are specific to one shell or the other get put into .bashrc and .zshrc. A nice effect of this is that when some utility I’m experimenting with decides to spam my login scripts, it’s another quick indication that my files have been trashed. Right now, my bashrc is only 16 lines long, and my zshrc is 41 lines, with nearly all my logic in commonrc. My other indication is that all these files are in source control, so I can tell by vcsh that something has been touched.

To handle local customization to my shells that I don’t want propagated, my .commonrc file ends up looking for and sourcing .extra and .extraextra if they exist. These do not go into source control. The reason for two files here has to do with $PATH. The first .extra file is designed to be the last thing to mess with that variable, then .extraextra can be whatever else is necessary.

The rest is a bit boring as the script sets preferences, always following a “if this tool is installed, prefer it, else look for the next tool” approach. So things always work, but after I install a bunch of my favorite tools, I’ll get more and more power. nix-configure gets nearly all my favorite tools installed, but nix is not worth bothering with on disposable cloud instances, for example, so sometimes I’ll apt install one or two tools I want, and let my rc scripts get them preferred. Basically, this is progressive enhancement adapted to the command line.

The last thing of note here is that there is a great set of zsh functions/helpers/ plugins/themes called oh my zsh that my rc scripts use. However, I found that oh my zsh almost works in bash as well. I have refactored this so it works on both shells, and use my repository instead.

How does this work? Part 3: Getting my normal programs onboard

So far, I’ve covered basic bootstrap through mr. This gets me access to any “unixy” terminal on Mac, Linux, BSDs, and even Termux on Android works after a few pkg install commands. I haven’t tried but it would likely work on Cygwin in Windows as well. This works well because everything is text files and shell scripts, at least up until now.

So…where do binaries come in? Today, this is through Nix. Before I extol the virtues of Nix, I first want to discuss my severe discomfort with the tool:

It is an OS (NixOS), a package manager, and a programming language. This causes a ton of confusion
The sands shift…a lot. I am not a heavy user, and generally when I ask a search engine or AI how to do something, probably 90% of the time, the nix CLI will tell me I’m using a deprecated way of doing it. I love that nix keeps old versions of its reference manuals around, but often times the first search result will take me to “the documentation”, but it’s an older version of the docs that again…leads me to deprecated ways of doing things.
Corollary to the above…there are two ways of doing things today between normal things and Nix Flakes. Flakes are super-useful, especially doing development. It’s stable, but requires one to add “experimental features” configuration to use.
It’s not kind to your machine. Nix package search is unusable without modern equipment, and I’ve had it consume all the memory on my low end equipment. Today, I consider nix search unusable (nix-search-cli works well, but requires Internet access). The nix store loves storage space as much as nix native search loves RAM, and it’s super-easy to download the Internet and thrash your CPU when you install new packages (more on this later).

With that out of the way, why do I use nix (the package manager…and as little of the programming language as possible)?

All the machines I use, I know I get the same version of the software everywhere
I have really good confidence everything will work
Development environments can use nix, and I really like this workflow
It handles much of the “find the right binaries for this system and wire it up”

This said, I do one “unnatural thing” with nix. Because NixOS doesn’t really like the way AppImages work, they instead extract and wrap AppImage files with a fairly involved process. I don’t really have time for that, I don’t use NixOS, and don’t really care to go through whatever work it would take to play nice. The AppImages are all controlled by hash anyway, so they’re still deterministic. With that in mind, I have a few flakes that avoid the wrapper, and just slap the AppImage into place. I’m sure purists hate this…but I’m a bit more pragmatic, and it works for me.

So…if I am going beyond “just get my shell configured in my way” on a machine, my next steps after bootstrap + mr up is:

Install nix (I use single-user, but I don’t think it matters)
Run nix-configure

nix-configure is a script that lands in ~/.local/bin, and by default installs all the things, subject to “I don’t want monstrosities copied down from the internet”. There is a command line option on the script to say, “no, really, download the Internet please”, and that will get all the things that require X Windows, etc. The difference between light and heavy is purely subjective. I’ve also recently learned about nix builders and substituters (cache), and have begun configuring them in. This, I hope, will help substantially with low-powered machines like setting up and using a Raspberry Pi. Once I feel more comfortable with this setup, it probably warrants a whole different blog post.

This works great for 95% of what I need to do. One sore spot for me is vim (actually, neovim). The neovim plugins I use always seem to throw various errors, because they’re not also locked into my system. This is completely doable, but I have not gone through the effort of solving it. Usually, I put up with the errors for a while, then get frustrated and try to figure out what broke. Usually, it’s a new change in a plugin that I haven’t bothered to update the corresponding config. Many times, it’s some combination of the following vim commands that fix the problem:

:PlugInstall
:PlugUpdate
:COQdeps
:TSUpdate
Lastly, waiting a while while nvim-treesitter trashes your machine compiling a lot of language files
Occasionally, :checkhealth tells me something useful

Treesitter is awesome, but walking up to new machines can be painful. It also is usually the thing that messes with me, and usually I end up finding some random breaking change they made in a GitHub issue or discussion, etc. For instance, just now prepping this post, I learned in the last two months, treesitter team decided to change the primary branch (but it’s not yet the default on GitHub), and changed the name of the configuration file that you need to require in lua. Both of these seem fairly arbitrary breaking changes, and both of which need detective work to track down why, once again, everything is just broken.

How does this work? Part 4: Software development

If you’ve read this far, and you have looked at my nix configuration repo, you may have noticed the complete lack of any programming languages being installed. I work in Zig a lot, and from time to time, I work with go, Java, Python, Rust, Node, or a host of other things. So…how does this work?

Enter mise. This is my newest addition, but it replaces a similar tool called direnv. They both do the same thing, but I think mise does it better. When I change directories, if mise sees a configuration file in that directory, it will adjust the paths, set up environment variables, or run Nix flakes (though that’s a little unnatural at the moment) to get everything set up the way that I need them for that project. Direnv just installed the tools, but with mise, I have it configured to just warn me if a tool isn’t installed, so I don’t have surprises if I’m just poking around, and I can choose to install with mise install (this just looks at the configuration and installs/uses the correct tools for the directory I’m in.

This works mostly great. One exception, is that I use vi for development, so if for some reason, I would rather use a graphical editor (usually something like VS Code, out of a Flatpak sandbox), well, this is a whole different problem. Flatpak has “SDKs” that can be installed and manually configured, but right now, it’s kind of a pain and it’s also too niche for me since I usually am at the command line using neovim. So I ignore it.

Overall

Is this perfect? Not by a long shot.

Updates sometimes require mr up (to update mr repo itself) followed by mr up (to update the vcsh repos)
Bouncing around machines often mucks with config, which then have conflicts I have to resolve on mr up
I need a recent (within the last 4 years or so) version of git installed by the bootstrap process due to my .gitconfig
Vim plugins (well, let’s be honest, 99% just treesitter) are a hot mess that I don’t have under control
AppImage considerations mentioned above
Nix still has a lot of downsides, and I haven’t even touched on the pain that I find Nix, the programming language
Mise + nix flakes are still emerging

But when it works, and actually, it works most of the time, it is pretty amazing. All my configuration and tools are there, ready for me, and I can be productive in less than a minute.

Knack backpack

Sun, 23 Mar 2025 11:32:00 -0700

Knack Series 2 Medium Expandable Knack Pack

I recently got one of these packs (limited edition pinecone color, no longer available), and I timed the order to take advantage of their 30 day trial policy, while giving it a serious run around the Earth, with travel planned from Portland to:

Seattle (overnight)
Seattle (day trip)
One week to India

As the bag is much more expensive than typical bags, I wanted to be really sure that it is a good bag. My March has has even more travel (add overnight to Manhattan and Singapore to the list above), so I’ve been taking notes while things are fresh in my mind. Of note, while I travel a lot (120k miles in 2024) and get great status benefits, my company is ridiculously cheap and my status does not get me upgrades on foreign carriers for free. My flights back and forth to India were in regular economy (so my comments below about seat space relate to economy rather than business or first).

tl;dr - this is a great bag, and worth the money

The Good

So…pretty much everything was good. The backpack is clearly very thoughtfully made, and I’m sure I haven’t even noticed 100 things I really like about the bag. Here is a short list of the things that have surprised me.

The whole backpack gets used: This is a bit weird, but one of my first thoughts was “wow, I’m just in and out of this backpack all the freaking time”. That sounds annoying, but I quickly realized, I was actually using the backpack. Every single thing I put in the backpack was getting used. My old bag was much bigger, and I had a bunch of stuff in there that just sat, unused, for the “just in case” scenario, which never came. Switching backpacks, I pulled stuff I hadn’t touched for a year or more.

The backpack is light: This is objectively true, but also, because I’m not putting extra stuff in the backpack, approximately half the time I pick up the bag, I’m worried I’ve left something behind because it’s so damn light.

The backpack strap pocket: OMG…this seemed so frivolous. Who needs, and who is actually going to use, that backpack strap pocket? Apparently - this guy! To be fair, it wasn’t until the trip to India where I realized its value. Day trip and overnight (even two nights) I didn’t need any other bag, so it was just me and this bag packed with clothes/toiletries in the luggage area of this bag, and it was permanently “a backpack”. When I went to India, it remained so until after the first leg of my flight, when I had it over the handles of my luggage and it occurred to me “I don’t need these straps…I can tuck them in!”. I thought I was so cool, and it looked nice. Then I got on the plane, and I realized, this isn’t just a gimmick. By tucking them in, sliding the bag under the seat in front of me was objectively easier - as it just slides underneath. Also, the straps weren’t spilling out into the aisle or my seatmate or the area under me at all. Afterwards, converting it back to a backpack was super easy.

Passport pocket: I didn’t even know this existed until I watched a video on how to pack the bag. Then I thought it was a bit gimmicky. But once I traveled internationally, that pocket is in the exact correct place to give access when you have it looped over luggage and are wheeling through airport security and customs and immigration and the thousand other times you need to show some officer or lounge person or hotel employee at checkin your passport.

The main compartment/computer sleeve: This was surprisingly large. I was worried about this, as overall the bag was much smaller than my old backpack. How would I put my stuff in there? Then I remembered…my computer does not go in that compartment! Having the computer in its dedicated place means a) I don’t need a sleeve, b) the bag doesn’t have to integrate a sleeve in the main compartment, and c) the computer is against an area almost guaranteed to be flat. It gave me a lot of confidence that I can stuff the bag pretty full and not worry about stressing the computer.

Overall size: I was really concerned about the size of the bag since I was ordering over the Internet. Would it always fit under the seat in front of me on planes? Would my computer fit? Would all my stuff fit? What about it being expanded? Nothing to worry about…I didn’t even need to expand the bag for my overnight trip, but later I did expand it just as an experiment. Even expanded, there was absolutely no problem having it in the seat in front of me. For the computer size, if I had a giant desktop replacement, this would be an issue, but my advice here is simply to follow the advice on the web site. My 14" was perfectly fine.

The Bad

None of my issues with the bag are minor. I had to think long and hard to come up with anything at all. This is all nitpicking.

Usage: So, a bit embarrassing, but I struggled with “where do I put my stuff”. Eventually I watched a video, but I can’t seem to find it. It might have been this: https://www.youtube.com/watch?v=44ovrl04tmE

Internal pockets: For my usage, the interior pocket layout is not 100% ideal. However, I can’t tell you exactly what I would change generally. Sometimes I feel like I want less pockets…sometimes it’s a feeling that I want more zippered pockets. It would be nice to have easier access to some pockets when in a horizontal configuration, as I do use it slipped over handles quite a bit. This is mostly non-specific feelings, but I do have 4 specific items though that I can pin down.

The bag has a “hidden pocket” that is really, really useful for passports. Transiting through foreign airports, you need access to your passport a lot, and it’s the perfect place and orientation for the job. However, this pocket should be maybe 1/4" wider (at least). Why? Because with just that much more width, it would be possible to put a paper boarding pass in there with the passport and zip it up. Going through international airports, I ended up with paper boarding passes a lot, and even paper lounge passes of the same size. I left the pocket unzipped and placed the boarding/lounge passes vertically so both they and my passport were available to various security checkpoints, lounge and boarding agents. Being able to put it in horizontally and zip it occasionally would have been nice.
I think they call this the “sunglasses pocket”. This should be a little bigger (more depth, maybe a touch wider) for my usage. In that pocket, I have a “hockey puck” wifi hotspot, earbud headphones, and my work badge. It’s a tight fit.
The laptop pocket. I struggled with this pocket for a while…my 14" laptop fits just fine in the pocket, but I really struggled to get the laptop in and out for a while until I realized I was doing it wrong. Seriously. It turns out that my hand was bumping into the attachment point for the backpack strap, and I was assuming the zipper stopped there, which meant that the opening I was trying to use is about an inch shorter than what it was designed for. Once I realized what I was doing, there’s plenty of room. Like I said, these are nits…I’m not sure what they could do about this, but it was a thing that happened.
The suitcase area hold down strap. I would like this to be more secure somehow, like maybe a zippered area around the whole thing similar to the other side. I suspect that particular suggestion won’t work due to expandability needs, but coming into the bag from the side there is a tendency to ruffle clothes with the current setup.

Slipping it over suitcase handles: I find while this works great, it tends to “hang up” a bit about half way down on the mechanism that allows my suitcase (an ebags Mother Lode roller board similar to this) handles to telescope. This could be as much of a problem with my suitcase as the bag (I have had similar, but less severe problems with other bags), but I add it here because I think I know what Knack can do to make this excellent bag better. My suggestion - on the edge of the back of the backpack furthest away from the side handle (the part that first goes over the handles when slipping it down), put some hard plastic such that it is the hard plastic that hits the handle joint rather than the softer edge of the backpack itself. This hard plastic could be along the entire edge, or maybe it is two pieces of hard plastic that users can attach and slide around, similar to the way the chest straps are designed.

The still unknown

Headphones + neck pillow: This combination proves a bit much for the bag. First, I tried attaching headphones to the loops at the bottom, but they would swing around when on my back, which felt weird, and it didn’t really work well when the bag was over the handles either. Then I realized that the ridiculously large main compartment (that doesn’t seem that large, but it is) happily handles the big over the ear headphones, while in their case, just fine. That left me with the neck pillow, which I strapped to the carry handle. It worked fine, and the pillow might have fit inside the main compartment if I hadn’t used the headphone case, or if I had put it into the suitcase area. I just don’t know.

Water bottle pocket: This is really cool, but I’m right handed, and would like it on the right side of the bag so I can more easily reach back while I have it on and grab stuff from that pocket. Also, it would be accessible when slipped over luggage handles. This would be in the bad category, but putting an engineer hat on for a second, I don’t see how it’s physically feasible while maintaining a) integrity of the bag, and b) allowing for the horizontal handle to remain. The horizontal handle is way more important in general usage. In my mind, I imagine that inside Knack, there were 4 hours of meetings over this very topic. One thing I learned about that pocket…if you don’t have a water bottle there, it’s a great quick place to throw wallet and cell phone when you’re going through security.

It seems to get flagged more by security?: However, I’m not sure if this is because I’ve been going through all kinds of different airports with varying security regulations (India literally wanted everything with a wire or microprocessor out of the bag), or if there’s something about the bag. It could very well be me, or my imagination. Domestically I think it’s only been pulled twice, but one of those times was coming through customs when the rules were normal (not PreCheck), and I left my computer in the bag.

The little loops at the bottom of the bag: I still literally have no idea what that’s for. I tried hanging my headphones off them, but it feels weird when something is swinging off them as you’re walking. I look forward to figuring that out.

Building Unikraft unikernels the hard way

Tue, 09 Jul 2024 14:14:48 -0700

Building Unikraft unikernels the hard way

You’ll note from my last post that unikernels have a number of advantages for building services, and Unikraft (the company) is making a run for a commercial offering based on the Unikraft open source microkernel.

Of course, they’re doing what they should be doing. Make a unikernel that can run linux binaries without modification. But that is dull and boring, and introduces a ton of overhead¹! Can we do better, but ignoring all that, and embracing the inner workings of the microkernel build system?

Can we do it with Zig?

Of course we can! Let’s go play

Getting started

First things first…let’s get hello world going. All this is easiest on Linux or macOS, and if using the cloud you need to make sure you have access to kvm, which may not be the easiest thing, nor the cheapest. Windows with WSL2 should also work, but I have only done any of this on Linux. Once you determine where you’re going to try this stuff, you’ll want to get QEMU installed, as well as kraftkit. Kraftkit has a few dependencies…if I remember correctly, I had most of what I needed on a default debian distribution except flex, bison and libncurses-dev (needed for kraft menu command). From there, you’ll want to clone the unikraft catalog.

Hello world!

The catalog is set up with a number of examples in the examples/ directory. THIS IS NOT WHERE WE WILL PLAY!. These examples are the boring, “build a regular Linux application and watch it work” stuff. We want the hard mode, advanced, expert stuff, which is located in the native/ directory. That’s where all the cool kids hang out, and we want to party there.

In that directory, there is a helloworld-c directory. Go into that directory, and do a kraft build and kraft run. Make sure to choose qemu/x86_64 when asked. I’ll wait. ;)

Once you’ve done that, you should see something like the following (I’ve modified the config, so the header will likely look different):

Learn how to package your unikernel with: kraft pkg --help
■▖   ▖■       _ _               __ _
■▚   ■▞  ___ (_) | __ __  __ _ ´ _) :_
▀■   ■▄ ´ _ `| | |/ /  _)´ _` | |_|  _)
▄▀▄ ▗▀▄| | | | |   (| | | (_) |  _) :_
 ▚▄■▄▞ ._, ._:_:_,\_._,  .__,_:_, \___)
                 Calypso 0.17.0~1a9f80b
Hello, World!

What the heck just happened?

ok, so we have a bunch of things that just went on. kraftkit did a bunch of magic for us, not all of which I understand quite yet, but here’s the high level:

Read the Kraftfile in the directory
Wrote a kernel configuration into .config.<appname>_<vmm>-<cpu arch> (which in this case means .config.helloworld_qemu-x86_64)
Pull the correct OCI images necessary to build the unikernel
Build the microkernel, using Makefile.uk effectively as an included Makefile from the main Makefile from the OCI image. Makefile.uk includes our helloworld.c
Take the build unikernel image, and lauch QEMU, specifying that kernel image should be used during the virtual machine boot

From a developer point of view, the important things to know right now are:

It begins with Kraftfile, but we can largely ignore that unless we want to upgrade the base code we’re building against
Our Makefile.uk file is pulled into the overall Makefile, so we can make changes to the build here
There’s a config file for the build

Unikraft is focused primarily on the “run stock off the shelf linux applications” use case, we need to be aware that documentation here is scant, so be prepared to dig, and also maybe read some source code. Originally, they did not have the ability to run Linux binaries, so the build system as described here is all they had. As a result, some material I’ve found, while useful, is also out of date. This presentation helped a lot.

Upgrading the base code

Let’s get a simple thing out of the way first. How do I upgrade to a new version of Unikraft? This was a problem I had in the last 3 months, as the version of Unikraft I was working with did not seem to play nicely with any version of QEMU that I tried. Luckily, there is a Unikraft version in the Kraftfile. So, I thought, the answer was “change that version number and rebuild”. Without detailed logging, it looked like it worked. It did not.

It turns out, that even after kraft clean, there is a cache of build artifacts, and the version specified in Kraftfile is ignored. The solution is to rm -rf .unikraft, and this will force re-reading Kraftfile and pulling a new version. Another problem is that arbitrary version shas as implied by the documentation is not actually possible. It appears as though the versions (or SHAs) you can use the ones specified in the unikraft manifest file at this link: https://manifests.kraftkit.sh/unikraft.yaml. This…took me a while to work out.

I haven’t done anything cool yet!

Well, you sort of have. But that aside, I get it. We have learned a bit about how all this works, and now, with that knowledge, we can get cracking. If you want to skip way ahead, you can look at my example repository.

The first thing I thought was “hey, I’ve now got a working application. Maybe the easiest approach is to compile a static library, and just link it in”. I like easy, so that’s what I tried.

First attempt was to fire up zig init on a subdirectory, add a line in build.zig to specify that we want to link with libc (lib.linkLibC), build it with zig build, and copy the static library from zig-out/lib into my directory. ok, this is a good start, but now I need to link it. How is that done? Well, I have a Makefile.uk, and that presentation from 2019 I found, and this gave me enough to stumble my way through. The trick is to add my library in the Makefile.uk like so:

UK_ALIBS-y += $(APPHELLOWORLD_BASE)/libziggy.a

My file was called libziggy.a, for no real good reason. After doing this, progress! And by progress, I mean a bunch of linker errors.

Seeing all the undefined symbols, I first noticed there were zig-specific symbols the linker was trying to resolve. Well, that won’t work, but I also know how to deal with that. Another edit to build.zig to specify that we want the compiler runtime built in (lib.bundle_compiler_rt = true), and we were on our way…to more failures. Specifically, we were now missing the following symbols:

mmap64
dl_iterate_phdr
getcontext
sigaction
write
close
realpath
read
msync
munmap
environ
openat64
flock
fstat64
dl_iterate_phdr
getenv
isatty

That is painful, but manageable. Later I learned that these are not all used at runtime, but they are referenced in the library, and so must exist. To address this, I spent a bit of time in the man pages, looking up function signatures and adding empty implementations for each of the functions above. You can see that work here if needed, or want to copy it in as you’re following along. You will also note that these aren’t really blank implementations. Each one will output “unsupported function called”, then force a segfault to make sure the VM crashes. That way we know what’s missing and can fix it.

After doing this, I needed a way to get undefined.c built and linked. We already know how to do this, we can just copy and paste into Makefile.uk:

APPHELLOWORLD_SRCS-y += $(APPHELLOWORLD_BASE)/undefined.c

With that…success! The kernel is built and ready to run. As always, my code is guaranteed to run the first time without any bugs. Especially “hello world”…

Why I can has no cheezburger?

…and, crash. But! The crash was not the terrible looping crash in QEMU that I had seen in the old version. We got past that after running their sample.

What was wrong, took me a while to figure out. I should note what the code I wrote actually does. zig init provides a library with an add function. I modified that function with a logging statement (uncreatively another “hello world”), and I called add from helloworld.c, which looks like this:

#include <stdio.h>

extern int add(int, int);

int main(void)
{
    int result = add(2, 2);
    printf("Hello, World! 2+2=");
    char buffer[10];
    for (int i = 0; i < 10; i++ ) buffer[i] = 0;
    buffer[0] = '0' + result;
    puts(buffer);

return 0;
}

The buffer stuff is dumb of course, but I didn’t want to mess with any additional libc functions or memory allocations or…anything. Just plain, dumb c outside of what I was testing.

Eventually, I noticed that if I compiled my zig library in any release mode (e.g. zig build -Doptimize=ReleaseSafe), it would actually work! And by work, I mean crash on my write() call in undefined.c. Without my logging statement, I could actually add 2 + 2 (I won’t spoil the answer here…you’ll need to compile you’re own unikernel and run it with QEMU to find out what that equals!), and not crash. So more progress, but clearly more to do.

I put my Debug mode crash problem on a shelf, and decided to compile in release mode for the time being. I know my library was eventually calling write, so I just needed to implement write(). This eventually turned out to be the wrong assumption, but I went down that path, pulled out my ancient c coding skills, and got fancy about it:

#define ANSI_COLOR_RED     "\x1b[31m"
#define ANSI_COLOR_GREEN   "\x1b[32m"
#define ANSI_COLOR_YELLOW  "\x1b[33m"
#define ANSI_COLOR_BLUE    "\x1b[34m"
#define ANSI_COLOR_MAGENTA "\x1b[35m"
#define ANSI_COLOR_CYAN    "\x1b[36m"
#define ANSI_COLOR_RESET   "\x1b[0m"

size_t write(int fd, const char *buf, size_t count){
    UNUSED(fd);
    UNUSED(buf);
    UNUSED(count);
    char *color;
    switch (fd) {
        case 1: // stdout
            color = ANSI_COLOR_RESET;
            break;
        case 2:
            color = ANSI_COLOR_RED;
            break;
        default:
            printf("write called on unsupported file descriptor: ");
            char fd_buf[10] = "fd = ";
            fd_buf[5] = '0' + fd;
            fd_buf[6] = 0;
            puts(fd_buf);
            SEGFAULT();
            break;
    }
    if (fd != 1) printf(color);
    if (fd == 2) printf("(stderr): ");
    printf("%.*s", (int)count, buf);
    if (fd != 1) printf(ANSI_COLOR_RESET);
    return count;
}

Once I worked through this, as long as I built with zig build -Doptimize=ReleaseSafe, I was chillin'!

I really can’t write to the console? How is helloworld.c doing this?

ok, so I know helloworld.c is writing to the console. They’re using puts, but console output is a clear use case, so it must be that Unikraft has solved this problem in the aggregate. What gives?

A bunch of research later, and some blog post, it might have been this guide, but I’m not 100% sure, and I started really poking through kraft menuconfig. Stumbling into their “Library Configuration” section, I struck pay dirt. posix-tty said “Support for stdin/out/err”, and when I turned that on, I got a build error. This…was a good thing, because specifically it was complaining about duplicate a duplicate symbol name for write(). I #ifdef’d my way around the build error, and got output! So, turning on the correct options in the Library Configuration section should eliminate my need for undefined.c entirely. This is still on my TODO list, or at least setting up the correct #ifdef statements and error messages is on my TODO list. I’d love to be able to take a default Unikraft configuration and just run zig code. Also, I like that my stdout shows up in red.

Loose ends

Two things were left to do. 1) track down Debug mode builds, and 2) compile and run “the zig way”, which is zig build and zig build run. For debug mode, the problem eventually traced to the libc call gettid(), which is implemented if you turn on the new posix-process stuff in kraft menuconfig. But zig does not call that, preferring the direct Linux syscall, and I haven’t figured out the Linux syscall shim in Unikraft yet. Zig does not call it, as zig supports glibc going back to a time when gettid() did not exist in libc. It only looks for the current thread ID when performing locks in Debug mode, which happens to happen when performing a log function call. So…that’s the root cause, but I don’t really have a reasonable path forward, other than asking other zig contributors what they think.

Compiling and running was much more straightforward. I will spare the details here, but my build.zig in the example repo has the details. Note that I’m not doing anything special to avoid dependencies on the system, so you still need QEMU and kraftkit installed. But I do have a (mostly) working zig build distclean build step that will also remove the cached .unikraft directory that really confused my debugging.

I’m kidding/not kidding on the overhead. In the grand scheme of things, unikernels eliminate so much traditional overhead, the small amount of extra time to parse and load elf binaries and provide the other compatibility is fine for most use cases. ↩︎

unikernels

Tue, 09 Apr 2024 17:15:30 -0700

Unikernels

I recently saw a notice on Hacker News talking about KraftCloud. I’ve seen previous news about UniKraft and Unikernels in general and have more than a passing interest in the technology. This news item had me take another look at the state of the technology, and what I learned was interesting.

What is a Unikernel?

Typically, an application runs on an operating system, which runs on hardware. Sometimes, the operating system is running in a virtual machine, which is supplied by another operating system (called a hypervisor), running on hardware. Sometimes, the application is running in a container on and operating system. Sometimes, containers and/or virtual machines can be nested. It all makes sense based on what you need to do, but this can get confusing, and is often unnecessary. For example, if I want a server that runs a microservice and calls other microservices, do I really need a full operating system that has everything needed to run the latest AAA game while watching streaming video and chatting with my friends? By eliminating everything besides what we need to run my microservice, we gain a lot:

Less storage for the operating system, application, and dependencies
Less memory usage as we lose all the stuff loaded “just in case”
Less CPU as we don’t have services we don’t need
Less attack surface for attackers to leverage
Faster initialization (boot)
Immutable deployment
Performance through the ability to tailor the OS to the needs of the application

We can achieve these benefits in a few ways:

Smaller docker images
Stripped down operating systems
Minimalistic hypervisors

But, there’s another way. What if we build our application directly into our operating system? This takes minimalism to the extreme, or logical confusion, depending on your view. This idea has been around since the mid-90’s, but the term “Unikernel” appears to have been coined in a 2013 paper.

Current Landscape

Since 2013, several research projects have been created around Unikernels. A good list can be found on GitHub. You’ll notice that on each kernel has a specific set of languages supported. This is a major drawback to most Unikernels (yes…that’s foreshadowing). Since the application is literally compiled with the kernel, the kernel and the application must be built together, at the same time, with a common set of tools. Typically, the languages are fairly low level, primarily because Unikernels must be written in languages that can speak to “hardware” (where hardware might be a virtual machine provided by KVM, Xen, VirtualBox, Hyper-V).

In my opinion, this has been the major drawback that has kept Unikernels in a niche space. Something for me personally to keep an eye on, but not a technology to get excited about.

When I started to revisit the topic of Unikernels when this news from Unikraft was announced, I took a peek, and noticed they are supporting “any” docker application with their Unikernel. How does this work, if the application is immutably bundled with the kernel?

I took a look, and there are three major Unikernel projects all trying to solve the developer experience and operational experience for Unikernels. OSv was first announced in 2014, took a break from 2015-2018, and has been active since. Nanos and Unikraft trace their roots back to 2017. All three have this magical ability to run multiple languages…so what gives?

Enter Docker

Docker has become enormously popular since its launch in 2013. Developers and operations appreciated the benefits of reproducibility and immutability of docker images. But Docker was interested in creating an open and level playing field, and eventually standardized, in 2017, the format used to create docker images. So, the industry had a standard format for reproducible images that could be used for Unikernels, and Docker also reinforced the trend toward Linux-based microservices.

Without talking to the people on these projects, my assumption is that, in 2017, a few folks had the idea to build a Unikernel with a standard application. That application’s responsibility would be to load and run Linux binaries. Linux binaries use a standard ELF binary format, access the kernel through a standardized set of syscalls, and with OCI, have a standard way of packaging the binaries.

All three have gone down this path, and in 2020, unikernels were split, from a taxonomy perspective, into “Language-based unikernels” and “POSIX-like unikernels”. Evenutally, researchers have said, “rather than create a Unikernel from the ground up with Linux support, why don’t we strip down Linux instead?”. Thus was born, in May 2023, the UniKernel Linux project. I could not find boot time measurements for this project, but the Firecracker project claims to have a custom linux kernel booting in 125ms. I’m a bit skeptical of the UKL’s project ability to compete on that particular metric as Unikraft, for example, claims boots in 6ms. My personal observations on Unikraft are that 6ms is, in fact, about correct.

Why, though, is it important to boot your machine this quickly? Well, the primary thing for me is that 119ms, or the difference between boot times between optimized Linux and, in this case, Unikraft, is also the difference between providing a microservice that needs to run 100% of the time and one that can simply boot up, serve a request, and then shut down completely…

Unikraft

All three of what I consider to be the major players in the Unikernel space offer the same features. All provide Linux binary support with an orchestrator that provides someone with a CLI that feels like docker. Compiling a native application (one that does not use the ELF loader and Linux syscall emulation) is left for the truly daring in each. OSv and Nanos do not provide directions for this at all, and while Unikraft does, I was unable to get a functional kernel on my machine, which was probably the same issue as reported here. Of the three, I’m most excited about Unikraft, for a few reasons:

They have a commercial business in the making, based on the core Unikernel technology. They need the open source kernel fast, compatible, and developer friendly to succeed. They have VC funding to pay people to make this happen, in addition to the open source community. The primary backer of Nanos sells command and control. The primary backer of OSv sells…nothing at the moment?
It seems like they have had a lot more momentum then the others.
They are still open to native apps, and their Linux binary support is actually a separate repo from the core.

On their commercial offering, I joined the beta, and I’m impressed. The performance was as advertised, the developer experience was great, and the “scale to zero” works well. I haven’t tried it myself yet, but “scale to zero” should also work with non-HTTP connections. Their documentation describes what is effectively a load balancer in front of your unikernel deployments, so a service is always there to listen for new connections and trigger your unikernel.

Is it ready to use? I would say no, not yet, at least not in a general sense. The idea here is to have Linux binaries “just work”, and without support for the fork() system call, we get pretty limited. If you craft your binary with the assumption of working in Unikraft, then yes. But that’s not their goal, and it really shouldn’t be ours. While I can’t find it on Unikraft’s roadmap, one of their core team has stated that fork is coming soon. I know I will be watching this space closely.

Neomutt setup

Sat, 13 Jan 2024 11:43:00 -0800

Neomutt

In my last post I mentioned I am using neomutt for email. This seems completely archaic…how can one do this in modern times? Wasn’t this built for pre-world wide web times, before HTML was even invented?

Well, yes, but it can still handle all the modern crazy, and for those not afraid of a terminal and keyboard, it has several advantages:

“Modern” mail clients don’t work in the terminal
“Modern” mail clients need to be driven by using the mouse. In fairness, both Outlook and GMail have copius keyboard shortcuts, but they are clearly mouse-first
“Modern” mail clients take a lot more memory. All my mail, starting from the ’90s, is available to me in neomutt. Memory usage is 339M (RSS). Opening gmail, once my fan spins down and my browser is responsive again, creates a tab that consumes 800M. Outlook on my work machine has 59 threads consuming 589M working set (833M private bytes virtual memory)
“Modern” mail clients include tracking.
“Modern” mail clients display html emails that include tracking mechanisms so marketers can determine open rates for email.

ok, so if these advantages are interesting, the question naturally becomes, “how on earth do you deal with email, 99% of which these days is html?”. This is a good question, and to be clear here, I’m only going to address inbound email. When I send email, it’s text only…I have need to format email probably less than once/year, even at work. If I ever have that need for personal email, I’ll likely just use nextcloud or something. I haven’t even run into the issue.

At a high level, this the combination of technology I use:

Mail client: neomutt
Terminal image support: This is provided through sixel support. I used to use st, now I use mlterm. They both have advantages/disadvantages. Honestly, I have yet to really find a terminal I’m 100% happy with. More full featured (but heavier weight) sixel-enabled terminals can be found at https://www.arewesixelyet.com/. Common terminals for normal people are xterm, iTerm2, and konsole. Also, tmux and Visual Studio Code both support sixels.
Terminal based browser: w3m. I’m not particular on this, it’s possible that lynx or something else might work…I just don’t do a lot of terminal-based web browsing here, I just want to view an email, but graduate to a normal browser for normal browsing.

For a while I had a basic setup going, but I re-visited recently and I now have a configuration I’m pretty happy with. A lot of the credit goes to this reddit post, and the information below is largely a re-post of the comment I made there.

With the software above installed, the configuration needs to be set up. ~/.mailcap is a configuration file that describes what should happen when a mail client sees various types of mail. This drives a lot of the behavior. I’ve created a mailcap file that looks like the following:

# The first one here is triggered when going in to view the attachments, then pressing 'm' on the text/html
# -sixel changes w3m behavior a bit. Without it, images will show up, but only when you do something to the screen
# having the option shows the images immediately
text/html;                         w3m -sixel -o auto_image=TRUE -o display_image=1 -T text/html %s; nametemplate=%s.html; needsterminal

# This second one is chosen by auto_view due to the copiousoutput tag
text/html;                         w3m -I %{charset} -T text/html -cols 140 -o display_link_number=1 -dump; copiousoutput

# The third, non-existent one is going in to view the attachments,
# then press 'V' on the text/html attachment, which opens in a proper browser

The comments above should be pretty clear, but this gives me the following behavior in neomutt:

<Enter> from the index, shows HTML through w3m, with links at the bottom. No images or color. This is fine for 90% of my needs, and is privacy respecting. This behavior is driven from neomutt auto view, which looks for the copiousoutput attribute, which needs to be the second entry in mailcap (otherwise it would be the default for everything, which I don’t want)
Going into the attachment list (v from the index) and using <Enter> or m on the text/html opens in w3m, in full glory, with all images. Oddly, I thought I needed m in this situation, and was surprised to find out <Enter> works. This will send tracking pixel data and whatever other crazy the sender is doing, but I have to take affirmative action to view email in this way.
Going into the attachment list (v from the index) and using V on the text/html will open in my actual browser, which seems necessary for a lot of unsubscribe links in particular for some reason

The last bullet (using V on the attachment list) is enabled through the following macro defined in my neomutt config (I use ~/.config/mutt/muttrc) as my configuration theoretically will work in either mutt or neomutt:

macro attach 'V' "<pipe-entry>iconv -c --to-code=UTF8 > ~/.cache/mutt-mail.html<enter><shell-escape>open ~/.cache/mutt-mail.html<enter>"

I also have confirm_qq false defined in .w3m/config. This simply prevents w3m from confirming I want to quit.

This seems pretty good so far and gives me a nice way to progressively handle my html mail:

90% of the time, I just hit <Enter>. I can see the link text, and I don’t send pixel tracking data
8%, go into attachment list and hit <Enter> to see images. At this point, I’m sending tracking data, which I’m consciously doing in a trade off for information I find more valuable
2%, go into attachment list and hit V because I really need to interact with this thing beyond w3m. Usually unsubscribe links, but it could also be newsletters with links to source material (w3m is usually good enough for this though)

That’s it! In the end, it’s not a lot of magic config, but it’s clearly tricky to work out. Also, this works well for how I work…others are looking for different user experience, and will probably not like this setup…

SMTP and email notes

Tue, 07 Nov 2023 00:00:00 +0000

SMTP and email notes

This is a general post to document what I’ve been learning while setting up my own email server. It’s been a long time since working with SMTP directly, especially with things like SPF, DMARC, and DKIM. A few general feelings of this process:

It’s not terrible! Running your own server is something that general Internet advice says DO NOT DO. But that advice, at least so far, seems wrong. I can send to gmail without issue, and get 10/10 on mail-tester.com
There’s a lot of bad advice on the Internet
The system is not magic
It’s pretty flexible
Security has improved a lot since the days of “telnet to port 25”!

Let’s start with security, because with all the spam out there, email kind of begins and ends there

Security

Let’s start with “telnet to port 25”. Port 25 is in use, but connections can be TLS encrypted over that port. From the server side, we can require it. Also, authentication of course. TLS on all the other ports, and encryption in transit is covered.

Connecting to other servers, services also check other things. In order of importance, I find quite a bit of poor advice on the Internet.

Reverse DNS, or PTR records

Unlike what I read before hand, these are critical. Basically, you can have a near perfect score, and nobody will deliver your email without this. Of course, it’s the owner of the IP range that needs to do this, so setting that up will vary based on the hosting provider you use.

DMARC, SPF, DKIM

Only DKIM is marginally a problem. SPF, and especially DKIM, provide indications of the integrity of the mail. DMARC is simply there to define your policy of what should be done should SPF or DKIM fail. That’s the high level, but a good first read on these technologies is here on Cloudflare.

DKIM is the most finicky, mostly because you have to have everything aligned for it to work. But, it’s basic cryptography. You store the public key in DNS, along with a “selector”, and your server software signs the email with the private key. The email header describes what other headers are part of the signature, so your email can be sent through 10 relays and it doesn’t matter at all unless one of them messes with those headers or the email itself.

The most fun thing IMHO is that selector. I was initially confused why that’s needed, but the answer is relatively simple. If you use multiple servers or services, you don’t need to pass multiple keys around. You can also have multiple private keys to do rotation. The server provides the selector in the DKIM header, so the domain lookup will find the right DNS record for that particular email. More on this later, but it makes things pretty flexible.

Hosting provider

This gets a bit tough. The big cloud providers all have various restrictions on using their servers to send email. I’m sure they want good IP reputation (see below), but I suspect other reasons exist and I don’t want to speculate too much. I believe DigitalOcean and Vultr will allow it, but ultimately I found George DC had a pretty good deal, and is fine with sending email, as long as it’s not for spam. I even submitted a ticket to update the reverse DNS and it was done in about an hour.

IP reputation

I haven’t had to worry about this too much, but I did notice two or three hits on IP reputation lists. They were old and it probably doesn’t matter, but I wanted to know how to clean that up. It’s both easy and painful. Easy, because you typically just need to go to the place that claims you have a nefarious IP address and click a button to delist. Painful because you have to do it at every place that reports your IP address, and many of those sites need you to establish an account. One required me to submit a ticket (which seems to have been handled without human intervention). Note that I did not try to use my home Internet connection for any experimentation - the Internet says that’s a really bad idea, and I tend to agree it’s probably not worth trying.

Software

I’m using docker-mailserver, which sets up most things for you. At the moment they seem to be transitioning some of the component software, but the docs were reasonable and it’s possible to make basically whatever configuration changes you need. They don’t hide their component software, so if you want to make changes, you’ll be digging into postfix, dovecot, etc. documentation. But, postfix in particular is a good battle tested server that is extremely flexible. Your mileage may vary, especially since this does not include any kind of web-based email interface. I don’t really need that, but if you do, there are other alternatives to consider like mailcow. I cannot vouch for anything else though - if I want web-based, I can just connect my nextcloud server’s mail app to it through IMAP.

Advanced terminology and use cases

I’m interested in something called “split routing”, where one mail server handles some email, and forwards the rest to another mail server. SMTP is actually very decentralized, though, so the idea of “this is your mail server” isn’t actually really a thing. Yes, there is a mail server that hosts your mail box, but really, anyone can do anything. The flexibility can be confusing, but it’s also pretty powerful.

One thing that I have really comes to terms with is the idea of an MTA, or mail transfer agent. Typically, we talk about mail servers, but technically, they are mail transfer agents. A mail comes in, and if the software knows that the mail is stored locally, that’s where it transfers it to. Otherwise, it determines the next place to go to. Whether that mail comes in from an IMAP connection, SMTP, or another SMTP server, is immaterial really. It just matters “where does this mail need to go”. In postfix, this means that all we need to do for “split routing” is define overrides in transport maps for emails we want to override. Literally…that’s it. The stuff we don’t override goes to where it’s supposed to go to. With a lot of help from LinuxBabe, I set up split routing, and it basically looks like this:

My DNS MX record is my server
I signed up with kingmailer.org for a trial. They tell me to go set up an MX record. I IGNORE that.
I set up a kingmailer@ email address on my domain, in kingmailer.
I set up a user@ email address on my domain, but only on my own server

Note, that at this point, kingmailer doesn’t know there’s another mail server, and does not know about user@ email address. Right now, the server that DNS points at, which will handle all email, has no idea of a kingmailer@ email address. Time to make it aware.

docker-mailserver has a mechanism for overrides, so this bit is specific to that software, but I simply dump a postfix-main.cf into the configuration directory, with a single line:

transport_maps = texthash:/tmp/docker-mailserver/postfix-transport_map.cf

docker-mailserver default docker compose file will mount a config volume in that location. The transport map itself is also relatively simple:

kingmail@example.com     smtp:[kingmailer.org]

Inbound emails will still be rejected though because there is no local account, so using docker exec -it mailserver setup add email kingmail@example.com makes quick work of that. All that script does though is add a line to the account configuration and ask postfix to reload.

…and…that’s it! Because I specify the server here (with square brackets to prevent more processing), my server will not bother looking at MX records, it will just send the mail to the right place. LinuxBabe recommends adding your domains into this map with a value of local, and I played around with that, but kept getting errors. Leaving it out seems to work just fine, and my assumption is that the default, at least in docker-mailserver, is not a local transport.

One thing I really wanted to do, but again, it goes against the general wisdom of the Internet, is get a unified inbox in my actual email client. I’ve gone back to mutt (actually, neomutt) which has the disadvantage of that HTML email is inconvenient, but also has a strong advantage in that HTML email is inconvenient. So many trackers, etc that are no longer automatically triggered. I’ve been working with neomutt over IMAP with gmail for over a year now, and rarely need the actual web version. “Clicking links” is something that’s not as pleasant as I would like, but all in all, I’m not sure HTML email was all that great an idea in the first place. ;-)

Mutt/Neomutt don’t really handle a unified inbox very well, so I figured out a way to do this by syncronizing my postfix data directly into my mail directory, thus violating not one, but two pieces of generally accepted wisdom. First, “you really don’t want to have two accounts in the same mail directory”, and second, “don’t use syncthing to synchronize a subdirectory of a synchronized directory”. The latter came as a warning message directly in the UI of the software, so that was a little scary, but I’m ok… more on this in a second.

The first advice was legitimate. My mail directories use isync (aka mbsync) to synchronize all my imap folders in Gmail to my local storage. If I’m going to keep the mail separated, I don’t want my postfix mail to get uploaded, so I needed mbsync to ignore the postfix mail. I also needed syncthing to ignore gmail-sourced mail. I have decades of email in gmail, and I do not need that on my VPS with only 40GB disk. For the first problem, I actually forked the isync source code at the latest release, then added a 3 line patch to look for a substring of the file name, and ignore it. As all postfix-sourced email ends up with the server host name in the file name (but Gmail does not), I could use that to let mbsync ignore my postfix data. On the syncthing side, I used the following ignore pattern:

??????????.*.????????????,U=*:?,*
dovecot*

That first pattern looks nuts, but I basically tried to scope to exactly what matched the mbsync stuff. The dovecot line is due to some dovecot accounting files (I’ve been saying postfix, but the files are actually managed by dovecot, which also handles imap).

The last issue is the syncthing warning about syncing a subfolder of a synced folder. This is ok in this circumstance. I’m sharing my overall mail folder with every device I want to use neomutt on (this allows me offline access when I’m traveling). However, just my inbox folder/subfolders is being synced to my mail server, and shouldn’t be shared anywhere else. Mutt/neomutt can happily think it’s working with a local mail directory, while everything I do is transparently synced back to gmail (for gmail-based email), or to my mail server (for email stored there).

Tools

I mentioned kingmailer a couple times. I’m sure there are a bunch of reasonable services, but kingmailer had a 14 day trial that did everything needed to get comfortable with the process.

Debugging DKIM (is my DNS updated properly? Am I loading the right private key? What about that bug in docker-mailserver where the config doesn’t persist properly? Is docker-mailserver using OpenDKM or rspamd?) was a bit painful. I found https://appmaildev.com/en/dkim pretty useful. It’s a bare bones site, but is not restricted to three tests a day the way mail-tester is. They also have SPF and DomainKeys tests, but I didn’t need them. For a more general test (everything, not just DKIM), I used https://www.gmass.co/analyze, which also avoids the mail-tester restrictions. Eventually though, you really need to test with https://www.mail-tester.com, because that seems to be the gold standard for determining proper setup. If you’re scoring 10/10 on that site, which you really should be, and your email isn’t going through, the problem is them, not you.

All inbound email is managed via MX records for the domain. I use Cloudflare for that.

Georgedatacenter, as mentioned above, has been good so far. Debian and docker provide the remaining software stack.

Results

After testing this process thoroughly, I’ve moved the lerch.org mail to use this set up. Now I have docker-mailserver handling my own email (inbound and outbound), and push the rest of my family off to gmail. If the server goes down for some reason, the fall back MX (mail server, mail exchange), remains gmail, so they will pick it up. Any email I try to send will fail until I reconfigure or bring my server up, but that’s ok by me.

It is interesting to see the connections from various servers coming through my new mail server. There sure is a fair amount of spam. But so far (only a couple hours, granted), I’ve yet to see any spam get to my inbox. I’ll post an update if this all turns out to be a terrible idea, but so far I’m looking at all this advice on the Internet “you don’t want to host your own mail server” and saying, “really? This doesn’t seem too bad.”.

If you’ve followed along this far, I assume you’re curious, so here’s a full diagram of my email setup. I still have some fine-tuning to do, but I’m pretty happy with it. One note - another piece of advice you see a lot is to use notmuch to allow easier searching. I find this less than optimal for two reasons. 1) your inbox in (neo)mutt becomes effectively read only, and 2) the search index is a massive single file, so synchronizing it is a bit painful. I instead 95% rely on (neo)mutt’s built-in search, which is nice and fast if you stick to what’s in subject or headers (just from?), and use a special command that performs a notmuch-based search on my server if I need something fancy. The server index is kept up to date as email comes in.

Zig build, and how I created a "universal lambda" function

Sat, 21 Oct 2023 00:00:00 +0000

`zig build` overview for zig 0.11.0, and how I created a “universal lambda” function

Having worked with zig starting in 0.9.0, the language, compiler, and build system have come a long way. While still lacking documentation and stability, it feels productive, and with the 0.11.0 release, I have stopped reaching for the latest master branch builds.

One of the properties of Zig, for both good and bad, is that it contains everything needed for development. That means that while Zig is a language, it is also a build system, and with 0.11.0, a package manager. In fact, some people ignore the language entirely, with the exception of what is needed to build using zig. Lacking documentation however, it has taken me a while to really get in depth in the build system. I sense others are in a similar position. This, therefore, is my attempt to provide an overview.

Note that this post covers a snapshot in time, and some areas of the build, and in particular details I will cover regarding the MVP package manager features, will change significantly over the next few releases.

Terminology

Let’s start with some terminology, which has changed in 0.11.0. Quoting from Issue 14307:

Package: A directory of files, uniquely identified by a hash of all files. Packages can export any number of compilation artifacts and modules
Dependency: A directed edge between packages. A package may depend on any number of packages. A package may be a dependency of any number of packages
Module: A directory of files, along with a root source file that identifies the file referred to when the module is used with @import
Compilation artifact: a static library, a dynamic library, an executable, or an object file

You’ll note that the word “package” and “module” in particular are very similar. A package has uses a hash for caching/download purposes. A module, however, is really interested on what you can import with the @import keyword. This will remain confusing when using 0.11.0, because even if you understand the difference, the terminology has not been fully updated in the standard library.

What happens at `zig build`?

With terminology behind us, let’s cover the mechanics of the zig build command. Conceptually, the following process is followed:

build.zig.zon file is read. Packages described in the file will be fetched if they do not exist in cache.
A build executable (literally named “build”) will be compiled
The build command will be run. The command will create a directed acyclic graph of “Steps” that must be executed to accomplish the end result(s). The steps will then be executed in what is termed a “make phase”.

This means that zig code is run at three stages in the build:

Compile time code during the compilation of the build executable
Run time code during the “build phase” while running the build executable
Run time code during the “make phase” while running the build executable

Step 1: Package fetch

Currently, step 1 includes package fetch. This process reads build.zig.zon, and I do not currently believe there is a way to override this name. Currently, this process eagerly fetches all URLs embedded in this file, then applies a hash algorithm to all files unpacked. We are in the MVP Package Manager territory here, so things are rather fragile. For instance, tarballs support a limited subset of capabilities, and the internal file layout is hard coded to strip 1 level of directories. If you’re using GitHub, this is just fine. For those pushing the limits, be aware of these limitations, and especially that most of these internals are likely to change.

Step 1: Package fetch…the big lie

Packages are packages, but with build.zig.zon, they are not just packages. They also become modules. The package name defined in build.zig.zon also becomes a magical module with a root source file identified as build.zig. This can cause confusion on terminology, but is also intended and super handy. It is also super confusing, because if I have a dependent package in build.zig.zon called “dep”, and that package exports a module called “dep”, I can @import("dep") in my build.zig, but unless I call exe.addModule or lib.addModule in my build.zig, the same @import("dep") in my actual code will fail. This is super-obvious if and only if you understand pretty much everything I’ve written above this paragraph.

As a side note, packages are stored in the global zig cache. On Linux, this is, by default, $HOME/.cache/zig. Inside that directory, a directory named simply “p” holds the packages. In there, you will see directories named by the hash that matches build.zig.zon. The reason all this is important, is that once in the cache directory, the hash is not verified again. This enables us to go muck around with the files in there without penalty…a very handy fact to know if you’re debugging some package’s behavior when it is used in a consuming application!

Because there is no way in the build itself (step 2 of our zig build procedure) to use a module, we are given a way here to access our dependency, or at least the build.zig of a dependency. This allows build time access to our packages, which we can then leverage to do such things as “reference a dependency, and use their build helpers to tweak or reconfigure our own build”. This enables some powerful capabilities, as I will show in my example below.

Step 2: Compiling the `build` executable

What’s in build.zig is pretty boilerplate. However, there is no main() function. If this is an executable, where is main()? This could be hard coded in the compiler, but zig finds a different way. You can technically override this with command line flags, though you don’t want to. Running zig build will create a build with the main entrypoint defined in build_runner.zig, which subsequently references your own build.zig.

Because zig code always has comptime also as a possibility, we end up with 1 comptime execution and 2 runtime passes logically (as described below). As an aside, comptime in zig must support all compilation targets. Zig does not want compilation to be OS or Processor specific!

Step 3: Running the `build` executable.

With the executable compiled, the last step is to run the build. Ultimately, this executes the code in build.zig. This code doesn’t actually do anything tangible, however. What it does do is provide a dependency graph, to let the build know that when a user provides a target/goal/step of, for instance run, we should find that run step, then back track to determine all the prerequisite steps needed (and in what order) to make the end goal work. Let’s look at a quick example. To follow along at home, simply zig init-exe and look closely at build.zig, lines 32-53:

    // This *creates* a Run step in the build graph, to be executed when another
    // step is evaluated that depends on it. The next line below will establish
    // such a dependency.
    const run_cmd = b.addRunArtifact(exe);

    // By making the run step depend on the install step, it will be run from the
    // installation directory rather than directly from within the cache directory.
    // This is not necessary, however, if the application depends on other installed
    // files, this ensures they will be present and in the expected location.
    run_cmd.step.dependOn(b.getInstallStep());

    // This allows the user to pass arguments to the application in the build
    // command itself, like this: `zig build run -- arg1 arg2 etc`
    if (b.args) |args| {
        run_cmd.addArgs(args);
    }

    // This creates a build step. It will be visible in the `zig build --help` menu,
    // and can be selected like this: `zig build run`
    // This will evaluate the `run` step rather than the default, which is "install".
    const run_step = b.step("run", "Run the app");
    run_step.dependOn(&run_cmd.step);

The comments are added by zig init-exe, but if you’re trying to get some stuff done, you may not have taken the time to really think about it. Let’s start from the top. zig build does not have any step targets. So, it will run the “install step” by default. It is the same as zig build install. You can actually put as many build steps as you’d like on the command line. zig build install uninstall run test is perfectly valid. What we show above, though, is the code for the run step. The run step itself does nothing put put ‘run’ on the help menu and makes it a valid target. Again, this does nothing. What actually makes it do something, is the very last line of code. That line of code connects the command line step to the command that actually runs the code. The code can’t run without the code being built, so we see that run_cmd.step.dependOn(b.getInstallStep()) to let the build system know that a) first we install (compile and put the binary in the output directory), b) then we run, then c) we “do nothing” but complete the run step.

Only after the graph has been completed, the build will determine (starting from the command line), which steps the user is requesting, and will use the graph to define an ordering. With zig 0.11.0, steps can now execute in parallel. The actual execution of these steps is considered the make phase and runs the required steps, in order, based on the user input. As an example, zig build implies the “install step”. But a common command would be zig build run, that has at its goal to perform the run step (which depends on the install step). Again, a user is not limited to a single end goal, so you can, for instance, specify zig build test run on the command line.

If you’re doing a lot of fancy build stuff, one frustration might be the use (or lack thereof) of modules in the build phase. The lib/exe addModule functions are for the compilation of the target code, not for the build itself! This is mostly ok, as a package has a module available at build (via build.zig), so defining a pub fn in build.zig can then reference whatever you want. However, it currently limits chained dependencies, so an @import of dep-of-dep in package dep is not possible in the main project unless dep-of-dep is also in the main project’s build.zig.zon. I am fairly confident this will be fixed (maybe while introducing other problems?) in zig 0.12.0.

An example: A “universal lambda function”

I have created a package universal-lambda-zig that leverages these features to allow minimal changes to a typical console “hello world” that can be run in multiple environments. This includes all zig targets, including WASM/WASI, AWS Lambda, my own home grown web server, and Cloudflare. These environments are all wildly different from each other. Consider:

console exes run main() front to back, then exit
WASM/WASI do the same as above, but are highly sandboxed, and have no threading
AWS Lambda runs functions based on a bootstrap executable that, when run, makes a web request to a server that issues instructions. In the universal lambda implementation, this bootstrap process calls the client code in process (no exec calls).
My own web server relies on discovering a function in a dynamic library, then calling it
Cloudflare relies on a JavaScript wrapper that calls a WASM/WASI file and marshalls the appropriate input/output/arguments/environment

By adding a reference into build.zig.zon, then calling a “configureBuild” function, the universal lambda project has a view to the build process, can add modules, etc. From the application’s main source file (by default src/main.zig), you can access the universal lambda module and register your handler. From there, the package and module can take over:

Console application and WASM/WASI

All the universal lambda package needs to do here is get pass control back to the application. The build:

Adds modules for use by the application

The module:

Reads standard input
Calls the handler
Writes handler output to standard output

WASM/WASI is just slightly different, because the module won’t compile by default due to the fact that std.http.Server.Response is used in the event handler context object. A little comptime magic is used to escape that problem.

AWS Lambda

The design of AWS Lambda is pretty unique. Here, we need to take the application’s handler and bundle it with an http client to deal with all the Lambda internals. To me, the service seems over-engineered, but I’m not on that team and I’m sure they have good reasons for all that. Anyway, this was my original goal, so it’s still full of hacks and learnings. The biggest of which is that the build portion is actually using the AWS CLI and not making good use of the make phase, at all. There are current package manager limitations with transitive dependencies that are preventing me from putting this into the ideal state. Also, AWS doesn’t yet universally support TLS 1.3, which is the only secure protocol supported by the http client in zig 0.11.0. I plan to re-address this with zig 0.12.0, by which time I believe both issues will be fixed. In the meantime, it’s hacky, relies on the AWS CLI, and is coded to not even show the options if you are running anything other than Linux (other POSIX OS’s like mac, *BSD, etc should work, but haven’t been tested). The build:

Adds modules for use by the application
Sets up steps to package, set up permissions, deploy, and run the function using addSystemCommand which creates steps in the graph to run system commands. This is how we get to the AWS CLI

Fundamentally, in lambda, bootstrap is still a command line executable, so we don’t need to do any magic there, but we do need to insert a whole http thing…

The module:

run in the module, which is called by the application, becomes 500 lines of code and tests designed to integrate our application to the AWS Lambda system. It spins up a loop, requesting data from a web server (endpoint is passed via environment variables). Full docs are here: https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html
When data is returned from Lambda’s web server, the module passes that data as event data to the application’s handler, and provide the raw response object as context
After the handler returns, we take the response and post it to the appropriate Lambda endpoint
Errors are caught and similarly posted to the Lambda error endpoint
Loop continues

Flexilib

The design of my personal web server is to allow a set of independently developed (but trusted) projects to plug in as a single process. The goal is to eliminate managing lots of processes and their associated memory requirements by allowing me to slap together a bunch of disparate microservices and run them together. There is no assumption of security boundaries within the system. It is similar to, but more flexible than, how Redis handles modules. This is a little different, because in this case, our package needs to modify the build much more substantially. The build:

Adds modules for use by the application
Changes the install output from an executable to a shared library (note - I think this is a mistake, and ultimately the package should probably add an additional build artifact. This would allow things like zig build run flexilib to work properly)
Changes the root source file for the build to its own flexilib.zig
Adds a module for the application code. flexilib.zig can then utilize that module
Adds code to be compiled into the library. The code exports the necessary flexilib functions. Those functions handle marshalling into more zig-friendly data formats (flexilib uses C calling conventions)

The module:

Doesn’t truly exist in the traditional manner at the moment. It really doesn’t do anything other than exist to allow the build to work
The request handler ignores the application’s main function and just looks for a function named “handler”. I think this could be fixed by providing a run function, calling the application’s main function on first use, then utilizing our own run function to simply set a variable for the handler. But this remains a TODO for now.

Cloudflare

There are a couple options for Cloudflare worker support. Cloudflare workers operate in V8 sandboxes (called “isolates”). Since we’re in V8, we’re firmly in web technology land. This gives us the following options:

Compile zig to Javascript. I don’t know if this has been done, but it seems…rough
Compile zig to wasm, provide a Javascript wrapper to wasm functions, and run the handler through this infrastructure. This…seemed interesting and I did consider it. Fundamentally, it’s not too different from what we’re doing with Flexilib above. Compile the code as a library, expose the application’s handler as a wasm export, then put some JS goo around it.
Compile zig to wasm/wasi, and use Cloudflare’s experimental wasi support to run it. This option was ultimately chosen, as it was much more straightforward. The main drawback here is the experimental tag.

With the option chosen, the implementation was nearly identical to just a straight WASM/WASI compile from our first foray above. The primary changes were in the build:

Adds modules for use by the application
Adds a Cloudflare deploy step to the dependency graph, which:
- Modifies a built-in wrapper Javascript index.js to refer to the build output’s wasm file
- Combines index.js into another wrapper that has the necessary wasi interface goo (leveraged from Cloudflare’s Wrangler and cloudflare wasi projects)
- Determines the appropriate Cloudflare account
- Uploads the worker, along with memfs.wasm, needed by Cloudflare’s wrapper and the build output wasm file to Cloudflare
- Enables the worker if needed (only necessary if the worker is being created)

This took quite a bit of work, but was straightforward in terms of a) understanding the apis and interactions and b) understanding the structure of various open source Cloudflare projects.

The module is identical to the console application above:

Reads standard input
Calls the handler
Writes handler output to standard output

Conclusion

While the zig build system and package manager are powerful, there remain some edge cases. I was pleasantly surprised how functional and flexible the system was. In terms of “MVP”, I’d say that the bar has been hit. There are two primary issues remaining:

Transitive dependencies, especially at build time. My Cloudflare build process for example, really should be another package. The problem is that I can’t really do a package of a package, at build time, without adding the dependency directly to the application code. Ultimately, the application’s build.zig.zon currently requires a dependency for the flexilib interface, which is the result of this issue and shouldn’t be needed. Zig 0.12.0 is addressing this
Pathing problems. Adding the package’s module for use of the application requires an “anonymous module”, with the source file residing in the package just fetched. One could argue that this should be a recognized use case, and it may be in the future. In the meantime, I need to know the location the package’s files without relying on knowledge of the zig cache or the hash of the package. Right now, there’s a bit of an ugly hack that triggers the build system to initialize the correct dependency, then pokes into the file system looking for a file we recognize in the dependency’s location. The hack itself is a bit less hacky in 0.12.0, but the whole thing makes me want to take a shower.

Outside these two issues, I find it amazing how much I was able to twist and turn the build and runtime behavior from code outside the application. Quite an MVP, and looking forward to 0.12.0!

For further reading on zig build system, take a look at: https://zig.news/edyu/zig-package-manager-wtf-is-zon-2-0110-update-1jo3

The article takes a slightly different angle than I have here. I have also heard that with 0.12.0, we are likely to get more documentation around the build system. Definitely looking forward to that!

Wrapping up our exploration: A mini shell

Sun, 07 May 2023 00:00:00 +0000

Wrapping up our exploration: A mini shell

This is part of an ongoing series:

Part 1: Exploring embedded programming with the Sipeed M0S with the BL616 microprocessor
Part 2: Getting to hello world with Sipeed m0s (BL616)
Part 3: Simplifying the tool chain: First steps
Part 4: Simplifying the tool chain: Wrap up
Part 5: Learning the SDK and USB protocol
Part 6: Wrapping up our exploration: A mini shell

Through previous efforts, I now have two USB devices. One for debug messages, and one for whatever we want. I don’t have any brilliant plans here…this is primarily an academic exercise to get knowledge in this world. So let’s put together the beginnings of a shell to see how things work out. Our todo list is nearly complete…the main question remaining is “What is this SDK doing for us? Can we do it ourselves?”

The answer seems to be, “abstracting the hardware interface”. Through the Bouffalo Labs SDK, we don’t need to worry about the memory mapped I/O addresses and interrupts coming from various devices. I had expected to need to do some assembly work, but as it turns out, using the SDK allows me to simply use C. The hardware specifics are coded into the SDK, while in Linux many (but not all) of these details are in the device tree

Most of the hard work is done, so let’s create a shell! Since the last post, I spent a bit of time on some niceties, implementing colorized debugging, making indentation consistent, and removing unneeded code. One thing I haven’t quite sorted out, however, is the use of ep_tx_busy_flag. Waiting until this flag is set to false seems to hang my interfaces. Not waiting, however, will result in intermittent glitches. The glitches are fairly uncommon, however, and this is just an example, so I’ve decided to ignore the problem for the moment.

One software/hardware addition to the niceties, however, is to use system uptime in the debugging logs. This proved interesting, as we can see the timing between operations when, for instance, delays are added artificially. This is done through bflb_mtimer_get_time_ms(), which simply reads from the appropriate timer memory address and converts microseconds to milliseconds.

To keep things in main.c, I created two function pointers in the newly renamed cdc_acm_usb_interface.c. One is to notify when the data terminal ready state changes. The other, for when data is received. We’ll use these in main.c.

First, we want to display a prompt when our data terminal is ready (here, through screen /dev/ttyACM0 2000000). This looks like the following:

void dtr_changed(bool dtr) {
  if (dtr) {
    debuglog("DTR enabled: requesting prompt\r\n");
    display_prompt = true;
  }
}

int main(void) {
  board_init();

  cdc_acm_init();
  debuglog("Initialized");
  dtr_changed_ptr = &dtr_changed;
  data_received_ptr = &data_received;
  while (1) {
    if (display_prompt) {
      /* We can't display directly on the dtr_enabled interrupt, must be on the
       * main loop. Without any delay, we will not see a prompt. But even 1ms
       * is enough
       */
      display_prompt = false;
      bflb_mtimer_delay_ms(1);
      output(prompt);
      debuglog("displayed prompt\r\n");
      curr_char = 0;
    }
  }
}

A couple interesting things here. At first, I tried to display the prompt directly in dtr_changed, but found that it just wasn’t possible, and nothing would be displayed until further interaction occurred. I ultimately decided that issuing I/O while in an interrupt handler was probably a bad idea, so I used a volatile flag display_prompt to signal to the main function that we were ready to go. Even there, I was unable to just display the prompt…a small delay was needed. At one millisecond, no one will notice an artificial display, so while bflb_mtime_delay_us is available, I chose not to use it. My debug log on /dev/ttyACM1 was invaluable to this process.

ok, so a prompt is now showing. What about processing actual commands? Here we have a few tasks:

We want to echo characters back to the terminal as they are typed
We want to process a command when the enter key is pressed.

Here’s how that looks:

void data_received(uint32_t nbytes, uint8_t *bytes) {
  /* I think we're getting an SOH after our output, but not sure why exactly */
  /* This if statement is a bit fragile (e.g. it doesn't cover SOH + data) */
  /* so we may need some further processing */
  if (curr_char == 0 && nbytes == 1 && *bytes == 0x01) return;
  /* if (nbytes == 1) */
  /*   debuglog("Received the letter '%c'. curr_char %d\r\n", *bytes, curr_char); */
  if (curr_char + nbytes >= 1024) {
    /* We will overflow - bail */
    debugerror("command too long");
    output("\r\nCOMMAND TOO LONG\r\n%s", prompt);
    curr_char = 0;
    return;
  }
  /* Process new data */
  memcpy(&cmd_buffer[curr_char], bytes, nbytes);
  raw_output(nbytes, &cmd_buffer[curr_char]); /* Echo data back to console */
  if (nbytes == 1 && cmd_buffer[curr_char] == '\r') {
    /* User hit enter, process command */
    output("\r\n");
    bflb_mtimer_delay_ms(1); /* There is a microsecond delay as well */
    cmd_buffer[curr_char] = '\0';
    debuglog("Processing command '%s'\r\n", &cmd_buffer[0]);
    process_cmd(&cmd_buffer[0], curr_char - 1);
    output("%s", prompt);
    curr_char = 0;
    return;
  }
  curr_char += nbytes;
}

The data_received function is assigned to the function pointer provided in our usb c file. I found that data was coming in prior to data terminal ready, and I suspect that this was some setup data coming through. I could easily log it with my debug functions at this point, but ultimately I just throw away the data when displaying the prompt in main. In this function we have few guard clauses to make sure we don’t overrun our command buffer. Next, we copy the incoming character into our buffer, then check to see if enter \r has been pressed.

If it has, we need to swing into action. Pressing enter should move to the next line, so we output the \r\n sequence to make that happen. To avoid glitching, we insert an artificial delay, then we set a null character at the end of the command. This shouldn’t be necessary, and ultimately I shouldn’t be assuming null termination (this is a buffer overrun exploit waiting to happen), but this is sample code, so I’ve not dealt with that problem. The null terminator does let me quickly log the command, which was helpful in development to track down the extra characters before DTR was enabled. Lastly, we call the process_cmd function to do whatever it needs to do before outputting a new prompt and resetting the index to our command buffer. Not too bad.

To process the command, I only implemented a quick and dirty echo:

void process_cmd(uint8_t *cmd, uint32_t cmd_len){
  int prefix_len = strlen("echo ");
  if (strncmp((char *)cmd, "echo ", prefix_len) == 0){
    raw_output(cmd_len - prefix_len + 1, cmd + prefix_len);
    bflb_mtimer_delay_ms(1); /* There is a microsecond delay as well */
    output("\r\n");
    return;
  }
}

This is primarily to give a feel to how we could start an implementation of multiple commands. Here we just look for echo and output whatever comes after, but we could just as easily read/write/list/delete files, start wifi, perform encryption, or whatever we need right here.

And with that, my exploration is complete, and my “final” code is here: https://github.com/elerch/bl616-usb-cdc-acm/tree/dd9bd918628b40b090eaa3396053e4726a6d3116 If I have something that calls for this neat little device, I may revisit, but in the meantime, it’s been fun exploring modern embedded development.

Learning the SDK and USB protocol

Mon, 01 May 2023 00:00:00 +0000

Learning the SDK and USB protocol

This is part of an ongoing series:

Part 1: Exploring embedded programming with the Sipeed M0S with the BL616 microprocessor
Part 2: Getting to hello world with Sipeed m0s (BL616)
Part 3: Simplifying the tool chain: First steps
Part 4: Simplifying the tool chain: Wrap up
Part 5: Learning the SDK and USB protocol
Part 6: Wrapping up our exploration: A mini shell

With our toolchain in place, it’s now time to actually do something real. In part 2, we we were able to get a hello world over USB by largely copying and pasting our way to glory. Now it’s time to actually understand what we copied and pasted, and augment it. Here’s our current todo list:

What’s the difference between TinyUSB and CherryUSB?
(optional) What is this library doing for us? Can we do it ourselves?
What is going on in cdc_acm_template.c Specifically what is the dtr stuff?
What is USB_NOCACHE_RAM_SECTION USB_MEM_ALIGNX?
Figure out logging output destination

And now that we’ve simplified our software stack, we’ll learn how to read data. However, my personal philosophy is that “it’s hard to debug what you can’t see”, so before reading data, let’s complete a bunch of our todo list and get some debugging capabilities along the way. Right now, we have a single channel, and I’d like to avoid using CPU debug features or JTag or anything…I want to live in the comfort of a single wire from my computer to the device. However, I know from my webcam that a single physical device can provide multiple virtual devices, so I should be able to do that. If I am able to provide /dev/ttyACM0 and /dev/ttyACM1, I can put debug information on one interface while my program works on the other. I can even turn my debug interface into a full blown control plane. But first things first. My objective today - get two interfaces recognized by Linux.

Getting two interfaces

First, I’ll take a look at my meta files. We learned a lot about the project structure, etc., in parts 2 and 3 of this exploration, so armed with that knowledge, we can go back though each of the files in my project and make sure they’re appropriate. It looks like my proj.conf has the following:

set(CONFIG_CHERRYUSB 1)
set(CONFIG_CHERRYUSB_DEVICE 1)
set(CONFIG_CHERRYUSB_DEVICE_CDC 1)
set(CONFIG_CHERRYUSB_DEVICE_HID 1)
set(CONFIG_CHERRYUSB_DEVICE_MSC 1)
set(CONFIG_CHERRYUSB_DEVICE_AUDIO 1)
set(CONFIG_CHERRYUSB_DEVICE_VIDEO 1)

What are those last 4 doing there? Certainly I’m using CherryUSB, and I’m working with the device library, and the USB CDC class. But I’m not doing anything with these other classes, so why are they being included? I feel pretty strongly that this is not needed, so a quick removal of the HID line and beyond is worth trying. We’ll remove those 4 lines, build, and flash and see if anything breaks. As it turns out, all is good, so we have less code to deal with, and a proj.conf file that now reads simply:

set(CONFIG_CHERRYUSB 1)
set(CONFIG_CHERRYUSB_DEVICE 1)
set(CONFIG_CHERRYUSB_DEVICE_CDC 1)

I also know that most of the USB setup work is in cdc_acm_template.c. The template part of this tells me it’s time to go looking at CherryUSB. In the demo directory are a bunch of other templates, including an interesting one that seems to indicate support for multiple interfaces. It looks incomplete, but does tell me that this is possible. We’ll bookmark this file for the time being, and seek to understand how USB handshake works.

Along the way I learn an interesting tidbit that WireShark can sniff USB traffic with the usbmon kernel module in Linux, but this seems overkill for what I’m doing. All the setup in this template keeps talking about descriptors, which leads me to start investigating USB descriptors. I found that lsusb has a verbose option to show the device descriptors, which let’s me look at how my web cam (which provides /dev/video0 and /dev/video1) advertises itself.

Eventually, armed with enough knowledge to search the Internet with the right keywords, I search for ‘USB device desciptor handshake’ and run across an absolute gold mine of information. Seriously. Go read that article right now and come back. It’s amazing. What isn’t clear to me after reading the overview is “can one hardware device present multiple USB devices (the root of the tree)?”, and if not, at what level are multiple virtual devices presented to the OS? The problem here is terminology. We speak about “USB devices” or maybe “USB virtual devices”, where in USB world, “interfaces” are the thing. Through some research I’m lead to believe that there is only one device descriptor for a physical device. Files in /dev (Linux device? Virtual device?) are based on interfaces. And as this is a communication device, there are actually 3 interfaces involved, which we can see by looking at lsusb -v -d ffff:ffff or by checking the source code of usb_cdc.h in CherryUSB. There is 1) an interface called an “interface association” to say, “hey, I have this thing, and this thing has multiple interfaces”. This is what triggers a file in /dev, or what I’ll call a “Linux device”. Then, for what we’re doing, we have 2 additional interfaces. The first of these is an interface for “inbound” traffic (device to host), and the second for “outbound” traffic (host to device). This seems a little surprising to me, because there are also endpoints, and it seems as though one interface with two endpoints would be ok?

Each interface has at least one endpoint, so for CDC ACM, we have an “IN” endpoint, an “OUT” endpoint, each attached to their respective interfaces. We also have an “INT” endpoint, which controls the device. As this USB device is designed for things like modems, my assumption is that it is meant to be where one would send the equivalent of AT commands. In any case, I don’t see a way to work with that endpoint directly and haven’t bothered digging to find out…it is necessary but fairly irrelevant to the goal here.

So, knowledge in hand, let’s run back to the source code of cdc_acm_template.c. I can see these lines at the top. Now that I’ve gone through the article, I am confident these are my endpoint numbers:

#define CDC_IN_EP  0x81
#define CDC_OUT_EP 0x02
#define CDC_INT_EP 0x85

So it seems I can just duplicate these constants. Let’s call this new interace our debugging interface, and add new constants:

#define CDC_IN_DBG_EP  0x83
#define CDC_OUT_DBG_EP 0x04
#define CDC_INT_DBG_EP 0x86

But a big, important question remains. What are those values, where did they come from, and why are they defined like that? When originally trying to get this to work, I put a TODO next to this to figure it out, and had chosen different constants than what is above. After an hour or so of wondering why things weren’t working quite right, I returned to my gold mine of information (you read that, right?), to see this explanation of endpoint addresses:

Endpoint Address:

Bits 0..3b Endpoint number
Bits 4..6b Reserved. Set to zero.
Bits 7. Direction 0 = Out, 1 = In (ignored for control endpoints).

Wow, ok. So this means we really can only have 16 endpoints? Even now this seems a little strange. Moreso because after some trial and (mostly) error, I found I cannot seem to have, for example, endpoint 0x02 (endpoint 2, direction out) and endpoint 0x82 (endpoint 2, direction in). So each in/out pair eats 2 of my 16 endpoints. This seems to conflict with what the multi-device demo in CherryUSB is setting up, but that file is not a complete working example, so I have more confidence in my own conclusion. My range is simply 0x0-0xF, everything must be unique, and the top nibble is 0x0 for Host->Device and 0x8 for Device->Host.

So…what’s next? Now we need to get these endpoints in our device descriptor. There is a line:

CDC_ACM_DESCRIPTOR_INIT(0x00, CDC_INT_EP, CDC_OUT_EP, CDC_IN_EP, 0x02),

We can clearly copy this line, but that’s not the whole story. Everything in the descriptor has lengths, and we’re adding our interface descriptors (one association and 2 interfaces as defined in this macro) to something. Going back to our gold mine of information, we can see all these interfaces are part of the “configuration”, which is basically a defined set of interfaces based on how you configure the device. We only have one configuration based on high power, but that configuration needs to be told we’ve got two things rather than one.

Looking at the configuration descriptor, we can see this:

USB_CONFIG_DESCRIPTOR_INIT(USB_CONFIG_SIZE, 0x02, 0x01, USB_CONFIG_BUS_POWERED, USBD_MAX_POWER),

And if we look at the macro definition, we see it is defined as:

#define USB_CONFIG_DESCRIPTOR_INIT(wTotalLength, bNumInterfaces, bConfigurationValue, bmAttributes, bMaxPower

We’re adding two more interfaces (the association is just an association, it is not it’s own interface…you can tell because our starting value is 0x02. So, we need to change our config descriptor init line to:

USB_CONFIG_DESCRIPTOR_INIT(USB_CONFIG_SIZE, 0x04, 0x01, USB_CONFIG_BUS_POWERED, USBD_MAX_POWER),

And add a new CDC_ACM_DESCRIPTOR_INIT:

CDC_ACM_DESCRIPTOR_INIT(0x02, CDC_INT_DBG_EP, CDC_OUT_DBG_EP, CDC_IN_DBG_EP, 0x02),

However, we’re not done! We need to also change our USB_CONFIG_SIZE variable, which states:

#define USB_CONFIG_SIZE (9 + CDC_ACM_DESCRIPTOR_LEN * 1)

We now have 2 CDC_ACM_DESCRIPTOR elements, so we need to change that to:

#define USB_CONFIG_SIZE (9 + CDC_ACM_DESCRIPTOR_LEN * 2)

That first parameter is the starting interface number. O and 1 were taken by our first device, so we’ll start at 2. The last number is the string identifier for the interface. I had tried to change it, but I don’t see where that value is surfaced anywhere in Linux, so I left it as is.

ok…now the structure is complete. Just below this are these buffer definitions, and we’ll want to add a debug_buffer:

USB_NOCACHE_RAM_SECTION USB_MEM_ALIGNX uint8_t debug_buffer[2048];

So now I feel it’s time to check off a few items from the todo list. This USB_NOCACHE_RAM_SECTION must be in CherryUSB, so let’s go search for it. It turns out this is device specific, but for most devices, this boils down to the addition of a compiler attribute to say the variable is non-cacheable. Here is an example:

#define USB_NOCACHE_RAM_SECTION __attribute__((section(".noncacheable")))

ok..this is totally fair. We’re using these buffers between hardware devices, so we do not want to cache this. The USB_MEM_ALIGNX is similar, but the attribute attached will specify the correct alignment for DMA (direct memory access), which also seems very relevant here.

Moving down the file, we’ll want a new endpoint busy flag, and we’ll want at this point to address what all this dtr_enable and ep_tx_busy_flag stuff is doing. But we can do this in a bit after we make sure Linux is registering two devices in the first place, so let’s move on.

We see these functions, which we can now tall are all callbacks from USB interrupts that arrive in CherryUSB, which are then passed to us. So any global variables here should be marked volatile. We’ll want to visit all of these in a bit.

Next up, endpoint structures. Well, those seem interesting. Let’s make sure our control key, c key, and v key are well dusted off, and practice some copy/paste, to result in this!

/*!< endpoint call back */
struct usbd_endpoint cdc_out_ep = {
    .ep_addr = CDC_OUT_EP,
    .ep_cb = usbd_cdc_acm_bulk_out
};

struct usbd_endpoint cdc_in_ep = {
    .ep_addr = CDC_IN_EP,
    .ep_cb = usbd_cdc_acm_bulk_in
};

struct usbd_interface intf0;
struct usbd_interface intf1;

struct usbd_endpoint cdc_out_dbg_ep = {
    .ep_addr = CDC_OUT_DBG_EP,
    .ep_cb = usbd_cdc_acm_bulk_out
};

struct usbd_endpoint cdc_in_dbg_ep = {
    .ep_addr = CDC_IN_DBG_EP,
    .ep_cb = usbd_cdc_acm_bulk_in
};

struct usbd_interface intf2;
struct usbd_interface intf3;

Well, that was pretty stupid…very brute force, but it will work. What else do we need? Well, it looks like a bit more copy/paste work in cdc_acm_init:

void cdc_acm_init(void)
{
    usbd_desc_register(cdc_descriptor);


    /* Add primary comms channel */
    usbd_add_interface(usbd_cdc_acm_init_intf(&intf0));
    usbd_add_interface(usbd_cdc_acm_init_intf(&intf1));
    usbd_add_endpoint(&cdc_out_ep);
    usbd_add_endpoint(&cdc_in_ep);

    /* Add debug log comms channel */
    usbd_add_interface(usbd_cdc_acm_init_intf(&intf2));
    usbd_add_interface(usbd_cdc_acm_init_intf(&intf3));
    usbd_add_endpoint(&cdc_out_dbg_ep);
    usbd_add_endpoint(&cdc_in_dbg_ep);

    usbd_initialize();
}

Nothing else seems relevant to the task of getting two interfaces (we don’t need them to actually WORK yet. So let’s build/flash/re-plug, and take a look at sudo dmesg:

[10078355.057650] usb 1-10.3: USB disconnect, device number 56
[10078356.841514] usb 1-10.3: new high-speed USB device number 57 using xhci_hcd
[10078356.966023] usb 1-10.3: New USB device found, idVendor=10b0, idProduct=dead, bcdDevice= 1.00
[10078356.966025] usb 1-10.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[10078356.966026] usb 1-10.3: Product: BL616 Bare Metal
[10078356.966027] usb 1-10.3: Manufacturer: Emil Lerch
[10078356.966027] usb 1-10.3: SerialNumber: 2023-04-19
[10078356.971473] cdc_acm 1-10.3:1.0: ttyACM0: USB ACM device
[10078356.972087] cdc_acm 1-10.3:1.2: ttyACM1: USB ACM device

The last two lines here indicate we’ve got two devices! Let’s get them actually working.

Making both devices functional

So, now we have two devices and 4 endpoints. We’ll focus entirely on the “IN” endpoints (device to host) that allow us to write hello world. We can debug via blinky lights, but that is painful, so let’s do everything through USB. I mentioned before I had an hour or two trying to work out the endpoint numbering scheme. This was a bit of an adventure in debugging that I think is worthwhile to discuss. But first things first.

We want to output now to 2 different Linux devices, so we’ll want to modify main.c. This part is easy, just insert a log() function in main.c from within our while loop, then implement that function in the template file:

main.c:

    while (1) {
        if (inx++ >= 2000){
          cdc_acm_data_send_with_dtr(write_buffer_main, data_len);
          log("dtr_enabled_true_callbacks:  . Write\r\n");
          /* cdc_acm_log_with_dtr(write_buffer_main, data_len); */
          inx = 0;
        }
        bflb_mtimer_delay_ms(1);
    }

cdc_acm_template.c:

void log(const char *data){
    /* memcpy(&write_buffer[0], data, strlen(data)); */
    /* write_buffer[9] = 0x30 + debug_val_1; */
    /* write_buffer[20] = 0x30 + debug_val_2; */
    int len = snprintf(
        (char *)&write_buffer[0],
        2048,
        "%d\r\ndebug u8 val 1: %d, debug val u8 2: %d\r\ndebug 32 val 1: %d, debug 32 val 2: %d\r\nsending to debug...\r\n",
        out_inx++,
        debug_val_1,
        debug_val_2,
        debug_val32_1,
        debug_val32_2
        );
    cdc_acm_data_send_with_dtr(&write_buffer[0], len);


    int dbg_len = snprintf(
        (char *)&debug_buffer[0],
        2048,
        "%d\r\ndebug u8 val 1: %d, debug val u8 2: %d\r\ndebug 32 val 1: %d, debug 32 val 2: %d\r\n(debug log)\r\n",
        out_inx,
        debug_val_1,
        debug_val_2,
        debug_val32_1,
        debug_val32_2
        );
    cdc_acm_log_with_dtr(&debug_buffer[0], dbg_len);
}

WHOA! What is all this? Well, this is my experimentation and debugging in raw form. Rather than just show you the golden path to make you think, “wow, he has it all sorted out”, I wanted to share both failures and successes so you can better understand how to progressively work through the problem.

Let’s start with main.c. I call log with a string, then proceed to ignore it when we get to the template.c file. At first, I had a lot of problems getting any output at all. This was due to the values I chose for the endpoint numbers, and what was going on in the machine was a “crossing of wires” that was primarily effecting the dtr_enable and ep_tx_busy_flag. In this state, I was able to get output on /dev/ttyACM0, but not on /dev/ttyACM1. I needed to know the status of these flags at various times, as well as the endpoint numbers coming through our callback functions usbd_cdc_acm_bulk_out and usbd_cdc_acm_bulk_in.

Not confident I had a standard library to work with at all, I left space in the string I used with log, then poked in my debug_val_1 and debug_val_2 variables as you can see in the commented comment above in cdc_acm_template.c. From other code I could tell memcpy was available to me, and I took a flyer on strlen (yes, I know there are buffer overflow things that can kill me here, but I’m debugging…leave me be).

After getting that working, I could see the problem, and I left myself some notes. Then the fact that memcpy seemed to be builtin rather than requiring some import was bugging me. I realized that we’re not really getting away from this compiler, which is basically gcc, so what other functions can I just use? A search brought me to this page on gcc built-in functions, and lo and behold, snprintf was available. With that function, I just needed to check and make sure that my cast wasn’t going to break anything, but it worked great. With my new found debugging powers, I was able to track down the endpoint numbering rules above, and finally determine the dtr_enable flag, knocking another item off our to do list. DTR is data terminal ready, which was a modem term and it was unclear whether and how that applied in the land of USB. Well, it turns out it works great. When the host connects (via screen or cat), Linux notifies the device that the data terminal is ready. The device, in turn, sends an interrupt, captured by CherryUSB and sent to the usbd_cdc_acm_set_dtr function. Rather than register this function like the other two, the CherryUSB library defines an empty stub function with a __WEAK attribute so a linked object with another implementation will take priority.

So, as long as the endpoints are set up properly, the function will be called when Linux opens the device, and logs will flow. However, we have two interfaces, so now we need two variables. An array would probably have been better, but I just used two (volatile) values:

void usbd_cdc_acm_set_dtr(uint8_t intf, bool dtr)
{
    /* Based on above init, intf = 0 is normal, intf = 2 is debug */
    if (dtr) {
        if (intf == 0) {
          dtr_enable = 1;
        } else {
          dtr_debug_enable = 1;
        }
    } else {
        if (intf == 0) {
          dtr_enable = 0;
        } else {
          dtr_debug_enable = 0;
        }
    }
}

The ep_tx_busy_flag was duplicated in much the same way. This is set to true when something is written to the interface, but the library being used is asynchronous, with the process ending in usbd_cdc_acm_bulk_in. I don’t fully understand what’s going on there, but I don’t believe I need to at the moment either.

Time for some code cleanup. The work in progress can be viewed in all its glory here.

Simplifying our tool chain: Wrap up

Wed, 26 Apr 2023 00:00:00 +0000

Simplifying our tool chain: Wrap up

This is part of an ongoing series:

Part 1: Exploring embedded programming with the Sipeed M0S with the BL616 microprocessor
Part 2: Getting to hello world with Sipeed m0s (BL616)
Part 3: Simplifying the tool chain: First steps
Part 4: Simplifying the tool chain: Wrap up
Part 5: Learning the SDK and USB protocol
Part 6: Wrapping up our exploration: A mini shell

“The next step should be comparatively easy. Swap out the remaining usage of binary blobs in the toolchain.” - me, last week

Well, that was NOT comparatively easy, not by a long shot. It was difficult enough that I had to ask myself, “so, what exactly is the goal here?”. To that question, I came up with the following answer:

Reduce disk space usage for all the bits needed to build/flash
Ensure all build/flash tooling is based on open source
Get a way to easily use the SDK without a bunch of downloads and setup on the developer machine

Of these three objectives, I can confidently say I’ve fully accomplished one of them, and partially accomplished another.

Last time, we managed an open source version of the flash tool. With this in hand, I assumed that we could use a different C compiler (different version of gcc, or maybe even clang) to compile. The logic was, if I could compile with clang, I could compile with zig, and I could write embedded code on the BL616 with zig, which would be way cool. I still have a plan to get there, but it clearly will not be easy.

What I found in the process is that this is likely possible, but probably not desirable. The version of gcc recommended by the Bouffalo SDK is the Xuantie T-Head Gnu Toolchain. This is a fork of another Risc-V collaboration project to enable Risc-V compilation.

Gcc itself, as well as clang, support Risc-V, so compilation is not a problem. However, the T-Head version supports the RISC-V p extension opcodes enabled on the BL616. Also, it supports specific tuning for the processor with a proprietary mtune option.

Undeterred, I plowed ahead, only to be met with changes that were made to system includes in the forked toolchain. I was starting to get into some really hairy stuff that I did not want to maintain. I needed a new approach.

That new approach would be “make sure that there are no binary blobs in the T-Head toolchain, compile it from scratch, and shove it in a docker container”. As it turns out, this was very possible and the results are in my github repo here. Compiling the code took over an hour, but at the end of the process I got a good base image.

The next step was to utilize this base image and load, modify, and configure the SDK, decompiled post processing tool, and decompiled flash tool from the previous work. At the end of the day, we have a docker image that can be used in place of make, built with nearly all open source.

Using this image will consume slightly less disk space (1.8GB vs 2GB), but we now have an easy way to use the SDK without a bunch of downloads. Unfortunately, we’re not purely open source. The WiFi and BLE drivers remain proprietary, but everything else is now open.

Using the image to create a project

The downside to containerizing the SDK is that we now, in our project, need to reference what I’ll call “magical things”. Generally I don’t like magic, but in this case, I’ll accept it. But in the process of working through all the kinks in the container images, I’ve gained an understanding of the auxiliary files in the project. Here’s a tour:

Makefile: make will call cmake to make Makefiles, which end up in a build/ directory. This Makefile is pure boilerplate and can be used from project to project with no changes. But the file must be there so the initial make command kicks off the chain.
CMakeLists.txt: This is a lot of boilerplate, but does specify the source files, include and main file for the project. It also specifies the project name. Generally will not be touched beyond an initial commit, except maybe to add source files.
proj.conf: This is included by CMakeLists, and is intended to set variables indicating which additional libraries we want to bring in. It is a bit dubious that something like CherryUSB, which is a different library and repository, is just included in this SDK, but I don’t want to re-think that right now.
flash_prog_cfg.ini: This is used by the flash program, specifying the entry address and a few other variables. It’s almost all boilerplate, but you the filedir variable will need to change from project to project.

Once these files and your source code is in place, we’re ready to build. As long as docker is installed, we can use the following command to build the project:

docker run --rm -t -v $(pwd):/build git.lerch.org/lobo/bouffalo_open_sdk:2f6477f BOARD=bl616dk CHIP=bl616

Once that is complete, we can use this command to flash to the device:

docker run --rm --device /dev/ttyACM0 -v $(pwd):/build git.lerch.org/lobo/bouffalo_open_sdk:2f6477f flash BOARD=bl616dk CHIP=bl616 COMX=/dev/ttyACM0

One permissions note: if using docker proper (rather than podman), build output will be owned by root. Using -u $(id -u):$(id -g) as part of the docker commands above will address that.

With this done, let’s get back to the actual code. The code from part 1, slimmed down by this exercise, can be found on GitHub, and because we’ll use this code moving forward, here is the link to the current commit: https://github.com/elerch/bl616-usb-cdc-acm/tree/797141a2cdc450c3834476124f483fc6c1741859

Simplifying our tool chain: First steps

Sat, 22 Apr 2023 00:00:00 +0000

Simplifying our tool chain: First steps

This is part of an ongoing series:

Part 1: Exploring embedded programming with the Sipeed M0S with the BL616 microprocessor
Part 2: Getting to hello world with Sipeed m0s (BL616)
Part 3: Simplifying the tool chain: First steps
Part 4: Simplifying the tool chain: Wrap up
Part 5: Learning the SDK and USB protocol
Part 6: Wrapping up our exploration: A mini shell

Before going any further, the fact that I now have a bunch of random binaries downloaded from the Internet on my machine, and 2GB of downloaded SDKs, I really want to deal with that issue.

First on the list…the Sipeed SDK can be removed. We’re getting hello world through the Bouffalo SDK, and while there are differences that we haven’t completely figured out, let’s remove 1GB of stuff from my machine until it’s needed.

Secondly, the flash software. This is closed source (but distributed in an Apache 2.0 licensed repo), but if we run strings BLFlashCommand-ubuntu on it, we’ll see that it’s clearly a pyinstaller executable. Anyone else more than a little concerned it is closed source for “security”? Anyway, time for some reverse engineering. We’ll get to (a similar implementation of) the the flash loader in Python, giving us a) source code we can examine b) smaller disk footprint, and c) access to more platforms. The MCU side of this was done using Ghidra, which was cool, but I think we can get better fidelity this way.

Reversing BLFlashCommand

Easy part. Go to https://pyinstxtractor-web.netlify.app/, and process the executable. We’ll get a zip file of the original pyc and system libraries. Now we can ditch the system libraries because we’ll just use the system interpreter and libraries.

Python decompilers are kind of fickle, so let’s figure out the EXACT version of Python that created these. It turns out, the first two bytes indicate that exact version. We’ll run od -x BLFlashCommand.pyc |head and get the following:

0000000 0d42 0a0d 0000 0000 0000 0000 0000 0000
0000020 00e3 0000 0000 0000 0000 0000 0500 0000
0000040 4000 0000 7300 00b8 0000 0064 005a 0164
0000060 0264 016c 015a 0164 0264 026c 025a 0164
0000100 0264 036c 035a 0164 0264 046c 045a 0164
0000120 0264 056c 055a 0164 0264 066c 065a 0164
0000140 0264 076c 075a 0164 0364 086c 096d 095a
0000160 0001 0164 0464 0a6c 0b6d 0b5a 0001 0164
0000200 0564 0c6c 0d6d 0d5a 0001 0664 0764 0084
0000220 0e5a 0047 0864 0964 0084 0964 0665 0f6a

That 0d42 converts to decimal 3394, which we can look up at https://github.com/python/cpython/blob/main/Lib/importlib/_bootstrap_external.py#L339 to find it was Python 3.7b5. This was the last of the Python 3.7 bytecode versions, so we know it was a production version of 3.7, and beyond that, it shouldn’t matter.

Decompilers are an inexact science, so let’s get to work. First, we need to distill to exactly what we need. We dumped all the dynamic libraries (on linux, everything *.so can go. The high level action plan is:

Remove all the standard library stuff, so we are at least focusing on additional code
Decompile the rest with Uncompyle6. Ignore all failures
Create a brand new directory
Copy BLFlashCommand.py into the new directory
Establish a Python virtual environment in the new directory and activate it
Run python3 BLFlashCommand.py and catch the error
Fix the error, go to step 6 until no errors

This is easier than it sounds, and ultimately I switched to decompyle3 and created a docker image to help. The errors, generally, will fall into one of the following categories:

Missing import: copy the py file from the corresponding decompiled directory
Missing import, that import is a library in PyPi: add a reference to requirements.txt, run pip install -r requirements.txt. Note that figuring out the right library is a bit tricky, but helping us is the version numbers in the decompiled source code
Import conflict: This happens when, for instance base.py and a directory base both existed and were copied over. We need to delete base.py. Python’s error messages were a bit confusing when this occurred in nested directories
Syntax error: Decompilation error. Ultimately, I found several decompilers and had to pick one that worked.

Eventually it ran as far as providing usage information, so I ran it with the same arguments as I saw out of make flash. This gave me a new error related to config files, which I located in the SDK directory and pulled in.

Next, we have a partially working flash command that successfully spins eating memory. Time to bust out some normal debugging skills and get to work. And now that I’ve found 3 decompilers, I can compare their output to locate potential sources of errors. To help me with this, I built a docker image with all three compilers. I just ran this on all relevant pyc files so I’d have versions to compare.

We’ll skip the gory details here, but a few hours later, I have a fully working, “open source” version of the BLFlashCommand flash command, located here: https://git.lerch.org/lobo/blflashcommand. The spinning and eating memory, for the curious, was a problem in the configuration parser. I ultimately threw away all the code there and replaced it with Python standard library stuff.

You can see the changes I’ve made to the decompilation output (including one file that was nearly completely removed and replaced with a small shim) in the change logs on that repo.

The next step should be comparatively easy. Swap out the remaining usage of binary blobs in the toolchain.

Getting to hello world with Sipeed m0s (BL616)

Sun, 16 Apr 2023 00:00:00 +0000

Getting to hello world with Sipeed m0s (BL616)

This post is part of a series.

Part 1: Exploring embedded programming with the Sipeed M0S with the BL616 microprocessor
Part 2: Getting to hello world with Sipeed m0s (BL616)
Part 3: Simplifying our tool chain: First steps
Part 4: Simplifying the tool chain: Wrap up
Part 5: Learning the SDK and USB protocol
Part 6: Wrapping up our exploration: A mini shell

If you’ve been following along, we have 3 to do’s left after getting to blinky with this device. These are:

What’s the difference between Bouffalo Labs’ SDK and Sipeed’s repo?
Avoid binary toolchains
Figure out what’s up with hello world

So far, we’ve ignored #1 and #2, and bypassed #3 on the way to get something happening on the device. Along the way, we’ve learned how to build code for the device, we’ve learned how to flash it, and where the serial console lives. This has generated a few more questions:

Can I just stick wires into these holes in the board?
Can I move to other GPIO pins?
Can I route communications directly through the USB port I’m already plugged into?

My style is generally, “let’s follow the easy path the vendor has laid out, then understand it deeply until I can go my own way”. But I like to stay as easy as possible. Easy here does NOT mean busting out a USB to TTL device. I have evidence in front of me to suggest that’s ultimately unnecessary. First, Sipeed did not add pins to the dock to support this…they made that hard. This suggests that it’s not “the way” they really wanted us to use that dock. Secondly, when I first plugged this device into my computer, I got both power and data, in the form of /dev/ttyACM0, and indeed that’s still the way we flash things on to the device. With that in mind, let’s tackle the question of “Can I route communications directly through the USB port I’m already plugged into”. My senses point me to the idea that this is the right direction.

So, first step, let’s consider that all this is very “M0S Dev Dock” specific. This suggests that I’m barking up the wrong tree, and rather than Bouffalo Labs' SDK, I should start looking at the Sipeed fork. Let’s go.

What’s the difference between Bouffalo Labs’ SDK and Sipeed’s repo?

First, we’ll git clone https://github.com/sipeed/M0S_BL616_example. In GitHub, this repo isn’t listed as a fork, but it is. It appears to have been cloned locally, then pushed up. And maybe with some cherry picks after the fact. Let’s figure that out.

$ git clone https://github.com/sipeed/M0S_BL616_example
$ cd M0S_BL616_example
$ git remote add upstream https://github.com/bouffalolab/bouffalo_sdk
$ git merge-base main upstream/master
fce6ce539e98e17f87aff39989cee8fa222499de
$ git log -1 fce6ce539e98e17f87aff39989cee8fa222499de
commit fce6ce539e98e17f87aff39989cee8fa222499de
Author: jzlv <jzlv@bouffalolab.com>
Date:   Wed Dec 21 20:33:14 2022 +0800

    [chore][cmake] move app target source into board

They diverged in December. But, I see other commits pulled into the sipeed version. For instance, commit f1e8545 in the Bouffalo SDK is the same as fd2377eb in the Sipeed version. Clear evidence that these were cherry picked commits, which will make this job a lot harder.

Looking more deeply, git reports 78 commits added to M0S and 76 commits added to Bouffalo. Looking with a diff program, we see 337 differences between the repos at the time of writing. Ultimately, we may need to put a pin in this one.

One interesting difference that sticks out, however, is the presence of a sipeed/solutions directory that includes a few directories, including a “usbd_cdc_acm_with_uart” subdirectory. Well this looks interesting. Let’s play with that. Without looking at the code yet, this looks like a USB Daemon (usbd) (UPDATE: this is wrong - more below) with Linux Communication Device Class (cdc) ACM. So intuitively I expect to load this code and see a /dev/ttyACM0 device pop back up. What it does from there? Who knows.

We’ll use our docker container and run make CHIP=bl616 BOARD=bl616dk. Unplug our device, hold the boot button and plug it back in. Release the boot button. From inside the same directory, we’ll make CHIP=bl616 BOARD=bl616dk, and see the following:

./../../../tools/bflb_tools/bouffalo_flash_cube/BLFlashCommand-ubuntu \
--interface=uart \
--baudrate=2000000 \
--port=/dev/ttyACM0 \
--chipname=bl616 \
--cpu_id= \
--config=flash_prog_cfg.ini
['./../../../tools/bflb_tools/bouffalo_flash_cube/BLFlashCommand-ubuntu', '--interface=uart', '--baudrate=2000000', '--port=/dev/ttyACM0', '--chipname=bl616', '--cpu_id=', '--config=flash_prog_cfg.ini']
[21:30:12.808] - Serial port is /dev/ttyACM0
[21:30:12.808] - ==================================================
[21:30:12.811] - Program Start
[21:30:12.811] - ========= eflash loader cmd arguments =========
[21:30:12.812] - serial port is /dev/ttyACM0
[21:30:12.812] - chiptype: bl616
[21:30:12.812] - cpu_reset=False
[21:30:12.814] - ========= Interface is uart =========
[21:30:12.814] - Bootrom load
[21:30:12.814] - ========= get_boot_info =========
[21:30:12.814] - ========= image get bootinfo =========
[21:30:13.068] - tx rx and power off, press the machine!
[21:30:13.068] - cutoff time is 0.05
[21:30:13.118] - power on tx and rx
[21:30:14.022] - reset cnt: 0, reset hold: 0.05, shake hand delay: 0.1
[21:30:14.022] - clean buf
[21:30:14.023] - send sync
[21:30:14.224] - ack is b'4f4b'
[21:30:14.254] - shake hand success
[21:30:14.755] - data read is b'010016060000010027928001319735cf0eb417000f758010'
[21:30:14.755] - ========= ChipID: b40ecf359731 =========
[21:30:14.755] - Get bootinfo time cost(ms): 1940.98583984375
[21:30:14.755] - change bdrate: 2000000
[21:30:14.755] - Clock PLL set
[21:30:14.755] - Set clock time cost(ms): 0.19287109375
[21:30:14.866] - Read mac addr
[21:30:14.867] - flash set para
[21:30:14.867] - get flash pin cfg from bootinfo: 0x02
[21:30:14.867] - set flash cfg: 14102
[21:30:14.867] - Set flash config
[21:30:14.869] - Set para time cost(ms): 1.668701171875
[21:30:14.869] - ========= flash read jedec ID =========
[21:30:14.870] - Read flash jedec ID
[21:30:14.870] - readdata:
[21:30:14.870] - b'c8601600'
[21:30:14.870] - Finished
[21:30:14.876] - Program operation
[21:30:14.876] - Dealing Index 0
[21:30:14.876] - ========= programming /home/lobo/bouffalo_sdk/M0S_BL616_example/sipeed/solutions/usbd_cdc_acm_with_uart/./build/build_out/usbd_cdc_acm_with_uart_bl616.bin to 0x000000
[21:30:14.878] - flash para file: /home/lobo/bouffalo_sdk/M0S_BL616_example/tools/bflb_tools/bouffalo_flash_cube/chips/bl616/efuse_bootheader/flash_para.bin
[21:30:14.878] - Set flash config
[21:30:14.879] - Set para time cost(ms): 1.68408203125
[21:30:14.879] - ========= flash load =========
[21:30:14.879] - ========= flash erase =========
[21:30:14.879] - Erase flash  from 0x0 to 0xc89f
[21:30:15.045] - Erase time cost(ms): 165.34619140625
[21:30:15.049] - Load 2048/51360 {"progress":3}
[21:30:15.053] - Load 4096/51360 {"progress":7}
[21:30:15.058] - Load 6144/51360 {"progress":11}
[21:30:15.062] - Load 8192/51360 {"progress":15}
[21:30:15.066] - Load 10240/51360 {"progress":19}
[21:30:15.070] - Load 12288/51360 {"progress":23}
[21:30:15.074] - Load 14336/51360 {"progress":27}
[21:30:15.078] - Load 16384/51360 {"progress":31}
[21:30:15.082] - Load 18432/51360 {"progress":35}
[21:30:15.086] - Load 20480/51360 {"progress":39}
[21:30:15.090] - Load 22528/51360 {"progress":43}
[21:30:15.094] - Load 24576/51360 {"progress":47}
[21:30:15.098] - Load 26624/51360 {"progress":51}
[21:30:15.102] - Load 28672/51360 {"progress":55}
[21:30:15.106] - Load 30720/51360 {"progress":59}
[21:30:15.111] - Load 32768/51360 {"progress":63}
[21:30:15.115] - Load 34816/51360 {"progress":67}
[21:30:15.119] - Load 36864/51360 {"progress":71}
[21:30:15.123] - Load 38912/51360 {"progress":75}
[21:30:15.127] - Load 40960/51360 {"progress":79}
[21:30:15.131] - Load 43008/51360 {"progress":83}
[21:30:15.135] - Load 45056/51360 {"progress":87}
[21:30:15.138] - Load 47104/51360 {"progress":91}
[21:30:15.142] - Load 49152/51360 {"progress":95}
[21:30:15.146] - Load 51200/51360 {"progress":99}
[21:30:15.147] - Load 51360/51360 {"progress":100}
[21:30:15.147] - Load 51360/51360 {"progress":100}
[21:30:15.147] - Write check
[21:30:15.147] - Flash load time cost(ms): 102.0771484375
[21:30:15.147] - Finished
[21:30:15.147] - Sha caled by host: 8161b63f09e30787bd4ff7c3a28be1aed67fdb18c94a21fc757b2d6d5a5501a5
[21:30:15.147] - xip mode Verify
[21:30:15.190] - Read Sha256/51360
[21:30:15.190] - Flash xip readsha time cost(ms): 42.87548828125
[21:30:15.190] - Finished
[21:30:15.191] - Sha caled by dev: 8161b63f09e30787bd4ff7c3a28be1aed67fdb18c94a21fc757b2d6d5a5501a5
[21:30:15.191] - Verify success
[21:30:15.191] - Program Finished
[21:30:15.191] - All time cost(ms): 2379.883056640625
[21:30:15.292] - close interface
[21:30:15.292] - [All Success]

ok. Now we can unplug it and plug it in normally (no boot button). Before we do that, I’ll issue sudo dmesg -w so I can watch the kernel logs. Here’s the result:

[9080153.452919] usb 1-10.3: new high-speed USB device number 55 using xhci_hcd
[9080153.581289] usb 1-10.3: config 1 interface 0 altsetting 0 endpoint 0x83 has an invalid bInterval 0, changing to 7
[9080153.581570] usb 1-10.3: New USB device found, idVendor=349b, idProduct=6160, bcdDevice= 2.00
[9080153.581575] usb 1-10.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[9080153.581579] usb 1-10.3: Product: Bouffalo CDC DEMO
[9080153.581582] usb 1-10.3: Manufacturer: Bouffalo
[9080153.583841] cdc_acm 1-10.3:1.0: ttyACM0: USB ACM device
[9080322.418367] usb 1-10.3: USB disconnect, device number 55
[9080326.010047] usb 1-10.3: new high-speed USB device number 56 using xhci_hcd
[9080326.138678] usb 1-10.3: New USB device found, idVendor=359f, idProduct=0000, bcdDevice= 1.00
[9080326.138684] usb 1-10.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[9080326.138687] usb 1-10.3: Product: USBD CDC ACM WITH UART
[9080326.138690] usb 1-10.3: Manufacturer: SIPEED
[9080326.138692] usb 1-10.3: SerialNumber: 2023030911
[9080326.141672] cdc_acm 1-10.3:1.0: ttyACM0: USB ACM device

IT LOOKS GOOD! What we can see here in the logs is a USB device being plugged in. This was listed as Product: Bouffalo CDC DEMO. This event was triggered when I plugged the device in while holding the boot button, allowing me to add the firmware. Next, we see that I’ve disconnected the device (USB disconnect, device number 55). Finally, when I plug the device back in, Linux sees a new USB ACM device, called Product: USBD CDC ACM WITH UART. Can we do anything with this?

screen /dev/ttyACM0 2000000 and…nothing. So - limited success. Maybe we should actually look at this code to see what it’s doing?

We’ll head into M0S_BL616_example/sipeed/solutions/usbd_cdc_acm_with_uart and take a look at main.c. Scanning the code, I see this function

static void uart1_init(void)
{
    uart1 = bflb_device_get_by_name("uart1");

    s_cdc_line_coding.dwDTERate = 2000000;
    s_cdc_line_coding.bDataBits = 8;
    s_cdc_line_coding.bParityType = 0;
    s_cdc_line_coding.bCharFormat = 0;

    bflb_uart_init(
        uart1,
        &(struct bflb_uart_config_s){
            .baudrate = (s_cdc_line_coding.dwDTERate),
            .data_bits = UART_DATA_BITS_5 + (s_cdc_line_coding.bDataBits - 5),
            .stop_bits = UART_STOP_BITS_0_5 + (s_cdc_line_coding.bCharFormat + 1),
            .parity = UART_PARITY_NONE + (s_cdc_line_coding.bParityType),
            .flow_ctrl = 0,
            .tx_fifo_threshold = 7,
            .rx_fifo_threshold = 0,
        });

    bflb_irq_attach(uart1->irq_num, uart_isr, NULL);
    bflb_irq_enable(uart1->irq_num);
    bflb_uart_rxint_mask(uart1, false);
}

This confirms the bit rate is appropriate, so our screen command is good. What is it doing exactly? Well, apparently that usb port is connected to UART1, so that’s useful information. In order to receive data, it will need an interrupt, so we see IRQ attach and enable commands. I’m not sure what the rxint_mask is at this point, but it looks like we’re turning off any mask. So that isn’t in our way.

The processor will need to know the code to call when the receive interrupt is triggered. So I’m guessing there is a “uart_isr” function somewhere in this code. ISR typically also stands for Interrupt Service Routine. Good signs all around, so I check, and sure enough, it’s the function just above our init function:

static void uart_isr(int irq, void *arg)
{
    uint32_t intstatus = bflb_uart_get_intstatus(uart1);

    if (intstatus & UART_INTSTS_RX_FIFO) {
        LOG_D("rx fifo: ");
        while (bflb_uart_rxavailable(uart1)) {
            char c = bflb_uart_getchar(uart1);
            LOG_RT("0x%02x\r\n", c);
            Ring_Buffer_Write_Byte(&uart1_rx_rb, c);
        }
        LOG_RD("\r\n");
        bflb_uart_feature_control(uart1, UART_CMD_SET_RTS_VALUE, 1);
    }
    if (intstatus & UART_INTSTS_RTO) {
        LOG_D("rto: ");
        while (bflb_uart_rxavailable(uart1)) {
            char c = bflb_uart_getchar(uart1);
            LOG_RT("%02x ", c);
            Ring_Buffer_Write_Byte(&uart1_rx_rb, c);
        }
        LOG_RD("\r\n");
        bflb_uart_int_clear(uart1, UART_INTCLR_RTO);
    }
    if (intstatus & UART_INTSTS_TX_FIFO) {
        LOG_D("tx fifo\r\n");
        bflb_uart_txint_mask(uart1, true);
    }
}

Looking at this function, we have a general structure of:

Get the status
Status can be one of three things: receive fifo, rto or tx fifo

I’m not entirely sure what these do. Receive fifo means “receive first in, first out”, so we see a loop to gather each character and add it to a ring buffer. After reading all characters, we see this code set the UART to RTS, or “ready to send”. That’s clearly the primary “receive some stuff over the line” path.

RTO…I need to investigate. A quick search tells me RTO is “Receiver Timeout”. Many timeouts in low level programming are fully expected, so we shouldn’t assume (yet) that this timeout is a bad thing. Scanning the SDK grep -ri rto and weeding out the FreeRTOS references, it’s looking like RTO is configured and it’s probably just a normal part of life here, but we do see an attempt to flush whatever characters are in the uart at that point and add them to the ring buffer.

TX_FIFO. This is clearly “we’re transmitting some stuff”. We set a mask here of some sort, but at this point, I have no idea what or why.

The other interesting thing here is the multiple LOG_D functions. This leads me to strongly suspect logging statements are sent out to JTag, but I haven’t tackled that yet either. My hope here is to not run down that rabbit hole.

Software, protocols and interfaces

We are now much more in software, and the fact that I’m looking at ring buffers makes me a lot more comfortable. Without completing my to do list, I know a way to get the SDK to create a USB ACM device in Linux, so now it’s just a matter of code. Let’s dig around more and see what we can see.

First, I’ll take a look at the main function from the same main.c file:

int main(void)
{
    board_init();
    uart1_gpio_init();
    {
        static uint8_t uartx_rx_buffer[2 * 512];
        Ring_Buffer_Init(&uart1_rx_rb, uartx_rx_buffer, sizeof(uartx_rx_buffer), NULL, NULL);
    }
    uart1_init();

    extern void usbd_cdc_acm_template_init(void);
    usbd_cdc_acm_template_init();

    LOG_D("start loop\r\n");
    for (size_t loop_count = 0;;) {
        extern volatile bool ep_tx_busy_flag;
        if (ep_tx_busy_flag)
            continue;

        size_t uart1_rx_rb_len = Ring_Buffer_Get_Length(&uart1_rx_rb);
        if (!uart1_rx_rb_len)
            continue;

        if (uart1_rx_rb_len < 512 && loop_count++ < 1000) {
            continue;
        }
        loop_count = 0;

        uint8_t data[uart1_rx_rb_len];
        size_t uart1_rx_rb_len_acc = Ring_Buffer_Read(&uart1_rx_rb, data, uart1_rx_rb_len);
        if (!uart1_rx_rb_len_acc)
            continue;

        LOG_D("acc: %u, uart1_rx_rb_len: %u\r\n", uart1_rx_rb_len_acc, uart1_rx_rb_len);
        for (size_t i = 0; i < uart1_rx_rb_len_acc; i++) {
            LOG_RD("%c", data[i]);
        }
        LOG_RD("\r\n");

        csi_dcache_clean_invalid_range(data, uart1_rx_rb_len_acc);
        ep_tx_busy_flag = true;
        usbd_ep_start_write(CDC_IN_EP, data, uart1_rx_rb_len_acc);
    }
}

If you’re running truly bare metal, you can’t ever return. When you return, the device has no more code to run, so it simply halts. So along with all the other “non-OS” examples, we see here an infinite loop. Usually this is done with while (1), but here they wanted to initialize a variable, so they went with a for statement with an empty evaluation expression (denoted by the empty space between those two colons).

First thing, we see a volatile variable. This is common with hardware. If a compiler sees a variable, it will happily optimize it in tons of ways to make your code faster and more efficient. However, memory is used a lot between threads or even by devices (using memory mapped I/O), to communicate. So the volatile keyword lets the compiler know that the value of this variable may change without the compiler’s ability to predict that change. Searching for ep_tx_busy_flag, I see hits in the src/cdc_acm_template.c file. My guess at this point is that there is an interrupt triggered when data is sent, and we use that and a timeout interrupt to turn the flag on and off.

It’s also at this point where I notice the Bouffalo Labs SDK has an examples/peripherals directory with some similar code. However, the Sipeed example I’m focused on at the moment is much more built-out.

The loop doesn’t do anything if a) we’re busy, b) we have received no data, or c) we either haven’t received a lot of data or we haven’t spent a lot of time waiting. This last check (if (uart_1_rx_rb_len < 512 && loop_count++ < 1000) ) isn’t too thrilling. The time for a processor to loop 1000 times will vary from model to model, and maybe even what it’s doing. We’re super low level though, so access to clocks is…complicated. In any case, we seek to understand at the moment, rather than go changing anything.

Assuming all those checks pass, we’ll actually read the ring buffer. There’s a bit of a race condition here, so there’s one final check to make sure we were actually able to read something. Then we log the heck out of everything…to JTag. There’s a final cleaning bit that I’m not sure about, then we issue usbd_ep_start_write, which I would expect to write the data back out (like an echo), but we didn’t see that happen…so I’m not sure. I’m also not sure what this ep part of all the names are. Another item for the to do list. But we’re close to actually writing some code rather than just reading it.

It’s late though, so I need to come back to this. I do want to actually write something, so let’s start with something simple. I found the manufacturer strings in src/include, and successfully changed them. Each character is 2 bytes, plus another 2 bytes for the the USB_DESCRIPTOR_TYPE_STRING. This allows me to calculate the length of the string. Changing the code in this file from:

    0x0E,                       /* bLength */
    USB_DESCRIPTOR_TYPE_STRING, /* bDescriptorType */
    'S', 0x00,                  /* wcChar0 */
    'I', 0x00,                  /* wcChar1 */
    'P', 0x00,                  /* wcChar2 */
    'E', 0x00,                  /* wcChar3 */
    'E', 0x00,                  /* wcChar4 */
    'D', 0x00,                  /* wcChar5 */

    0x16,                       /* bLength */
    USB_DESCRIPTOR_TYPE_STRING, /* bDescriptorType */
    'E', 0x00,                  /* wcChar0 */
    'm', 0x00,                  /* wcChar1 */
    'i', 0x00,                  /* wcChar2 */
    'l', 0x00,                  /* wcChar3 */
    ' ', 0x00,                  /* wcChar4 */
    'L', 0x00,                  /* wcChar5 */
    'e', 0x00,                  /* wcChar6 */
    'r', 0x00,                  /* wcChar7 */
    'c', 0x00,                  /* wcChar8 */
    'h', 0x00,                  /* wcChar9 */

Allows me to see the new Manufacturer name “Emil Lerch” in dmesg and lsusb:

Bus 001 Device 063: ID 359f:0000 Emil Lerch USBD CDC ACM WITH UART

With this first change we’ve demonstrated at least a beginning of understanding. And a stumbled-on breakthrough.

For this simple task, we need to know the size of USB_DESCRIPTOR_TYPE_STRING, but that seems really elusive. I do see a definition in some cherryusb directory, but that seems like another example. After compiling, I see a binary match in libcherryusb.a in our build directory! This is the danger of my random spelunking through this process…I haven’t looked at what steps are being performed during the build yet. But clearly we’re building and linking some cherryusb library. Now, this SDK seems really mashed together, so when I do a search for cherry usb, lo and behold, we run into this: https://github.com/sakumisu/CherryUSB This is awesome, and it has docs! Once again, not in my native language… :( But code is universal, so we will continue to plow ahead.

Doing a bit of research, there is a random post on Reddit saying CherryUSB might be a fork of TinyUSB. Looking at the repos, they do seem similar, but with different MCUs supported. But most importantly to me, there is documentation in my native English language. We will add “figure out differences between TinyUSB and CherryUSB” to the to do list, and soldier on.

Next up, let’s clear a few recent items off the to do list. We’ll figure out that “ep” question. Looking at cdc_acm_template.c, we’ll see ’ep’ stands for endpoint. It’s how we’re setting up the “callbacks” for folks used to high level programming, or “ISR” for low level. We should also probably understand a bit more on how USB works. We are creating a USB “device”…aha…this is the d in “usbd”…daemon is clearly too high level a term for what we are doing. The device is USB class CDC, with a subclass of ACM. From there, we’re on our own. With this, my mind drifts back to examples/peripherals/usbdev. What’s in there?

It looks like there is a usbd_cdc_acm there. I wonder if that is an even simpler example. If so, we could look at the differences and get a better understanding of what Sipeed has done. Let’s take a look at that code:

int main(void)
{
    board_init();

    cdc_acm_init();
    while (1) {
        cdc_acm_data_send_with_dtr_test();
        bflb_mtimer_delay_ms(500);
    }
}

Well, that’s dead simple! We send some data, then wait 500ms. What are we sending? This cdc_acm_data_send_with_dtr_test function isn’t in this file, but it is in cdc_acm_template.c:

void cdc_acm_data_send_with_dtr_test(void)
{
    if (dtr_enable) {
        ep_tx_busy_flag = true;
        usbd_ep_start_write(CDC_IN_EP, write_buffer, 2048);
        while (ep_tx_busy_flag) {
        }
    }
}

We need to find out what’s in write_buffer, which is initialized slightly above this function:

void cdc_acm_init(void)
{
    const uint8_t data[10] = { 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x30 };

    memcpy(&write_buffer[0], data, 10);
    memset(&write_buffer[10], 'a', 2038);

man ascii tells us this is “1234567890” followed by 2038 “a"s. Well, that’ll be pretty obvious. Let’s run this one. make CHIP=bl616 BOARD=bl616dk works, and I can load it with make flash CHIP=bl616 BOARD=bl616dk COMX=/dev/ttyACM0. It works! And wow, that’s a lot of output. But it’s not “Hello world!”, and we haven’t written any code yet, so that’s just not acceptable. But this is simple enough, let’s get writing.

Hello world! Finally

Quick recap of our understanding at this point. We’ve identified that there is a simple Bouffalo Labs example for a USB Device. This device implements the CDC ACM protocol with the help of the CherryUSB library (similar to TinyUSB). The example outputs data, so we can use it for our example. Time to write code…finally!

The example has a fairly boilerplace cdc_acm_template.c file that they choose not to touch. It’s still doing some plumbing work, some of which we don’t understand yet. For instance, what is “dtr”? Seems like another to do list item, but we don’t need to understand it yet.

For this Hello World, I’d like to avoid touching or understanding that file too much, but I’d like an easy way to pass my own data from main.c to it to display. Currently, we can’t do that. So let’s copy the function cdc_acm_data_send_with_dtr_test and make a new one that can accept the data to display:

void cdc_acm_data_send_with_dtr(const uint8_t *data, uint32_t data_len )
{
    if (dtr_enable) {
        ep_tx_busy_flag = true;
        usbd_ep_start_write(CDC_IN_EP, data, data_len);
        while (ep_tx_busy_flag) {
        }
    }
}

Easy enough - I’ve simply added data and data_len parameters and used those instead of the globals in the file. Now let’s setup our own write buffer in main.c and declare the new function at the top of main.c:

extern void cdc_acm_init(void);
extern void cdc_acm_data_send_with_dtr(const uint8_t *, uint32_t);

uint32_t buffer_init(char *);

USB_NOCACHE_RAM_SECTION USB_MEM_ALIGNX uint8_t write_buffer_main[2048];

Now, I have no idea yet what USB_NOCACHE_RAM_SECTION USB_MEM_ALIGNX is. Clearly we’re telling the compiler this isn’t a cacheable variable, and it has special alignment requirements, but what exactly? No idea. Another to do list item.

Eventually, it would be nice to have a printf-like function, but we don’t have time for that. We’ll get part of the way there with an initialization function that takes a C string and copies it to our buffer. We can’t use the string directly, because it’s not USB_NOCACHE_RAM_SECTION USB_MEM_ALIGNX:

uint32_t buffer_init(char *data) {

  uint32_t data_len = 0;
  for (ssize_t inx = 0; data[inx]; inx++) {
    write_buffer_main[inx] = data[inx];
    if (data[inx]) data_len++;
  }
  return data_len;
}

We return the data length because it’s pretty easy to find here, so let’s save the manual counting. And with that, we can change main() to a more proper ‘Hello world!\n’

int main(void)
{
    board_init();
    uint32_t data_len = buffer_init("Hello world!\r\n");

    cdc_acm_init();
    while (1) {
        cdc_acm_data_send_with_dtr(write_buffer_main, data_len);
        bflb_mtimer_delay_ms(2000);
    }
}

make CHIP=bl616 BOARD=bl616dk and make flash CHIP=bl616 BOARD=bl616dk COMX=/dev/ttyACM0 later, reinsert the device, run screen /dev/ttyACM0 2000000 and we see the following!

Hello world!
Hello world!
Hello world!

Note that I changed the delay, so we see a new message appear every 2 seconds. The completed code so far can be found here: https://github.com/elerch/bouffalo_sdk/tree/e790f5fa86c40f2a788c78a5dbdec0ccfacf6209/examples/peripherals/usbdev/usbd_cdc_acm

Our current to do list:

What’s the difference between Bouffalo Lab’s SDK and Sipeed’s repo?
Avoid binary toolchains
What’s the difference between TinyUSB and CherryUSB?
(optional) What is this library doing for us? Can we do it ourselves?
What is going on in cdc_acm_template.c Specifically what is the dtr stuff?
What is USB_NOCACHE_RAM_SECTION USB_MEM_ALIGNX?
Figure out logging output destination

Next task? I think we have a choice. Either:

Learn how to read data
Simplify our software stack

For now, we’ll enjoy a device that can print Hello World without any debugging hardware.

Exploring embedded programming with Sipeed m0s (BL616)

Wed, 12 Apr 2023 00:00:00 +0000

Exploring embedded programming with the Sipeed M0S with the BL616 microprocessor

Note: I do not use Amazon affiliate links. The Amazon links below do NOT kick anything back to me.

This is part of an ongoing series:

Part 1: Exploring embedded programming with the Sipeed M0S with the BL616 microprocessor
Part 2: Getting to hello world with Sipeed m0s (BL616)
Part 3: Simplifying our tool chain: First steps
Part 4: Simplifying the tool chain: Wrap up
Part 5: Learning the SDK and USB protocol
Part 6: Wrapping up our exploration: A mini shell

I have been dabbling in low level programming lately. Late last year, I put together a Hello World assembly language sample in multiple ISAs to see the difference between various CPU architectures. To create something a bit more realistic, this included the use of Linux syscalls, so these samples work but require Linux. The advantage here is that I was able to understand better how programs actually interface with the OS (as long as the OS is Linux :) ). This repository is here: https://github.com/elerch/assembly-samples

I also created a zig-based program to interface with a small OLED display I got for Christmas. This was pretty fun, and allowed me to use some new features in the newest unreleased version of zig to automatically download/build ImageMagick and compile it into a single static binary that can work with these devices. There’s lots more that can go into this program, but the exercise really was about getting closer to the hardware/software interface. It has literally been decades since I’ve had to cross reference data sheets with my code and worry about things like clock dividers, etc.

Most recently, I read the book Code, which reminded me a lot about my undergrad classes at Lehigh, and refreshed my memory on how hardware/firmware/software interact. Indeed, it went beyond some of the classes I had had, and overall it was a very pleasant read. Highly recommended book, although it gets pretty dense, so if you’re not super into this stuff, you might want to read until your eyes glaze over, then skip to the last three chapters or so where things get a bit higher level again.

Some of the trigger for this is my recent interest in Risc-V. I’m pretty excited overall for this instruction set architecture (ISA). Wikipedia has many more details, but this is an ISA that was developed at University of California, Bekeley to be an open source ISA. The ISA describes the interface between software and hardware for CPUs. So now there is an open standard interface, and hardware designers can create either open or proprietary CPU designs that adhere to that interface. An open ISA frees developers from specific vendors that may have business issues or come under political pressure. Because of this, RISC-V is already wildly successful, though only in the embedded world. That said, the possibility of RISC-V phones, tablets, and computers are on the horizon.

With this interest, I have been experimenting with real hardware. My first experiment was using Debian on the Sipeed LicheeRV Dock and shortly thereafter on the MangoPi MQ Pro. These are both based on the Allwinner D1, and are therefore nearly identical. This was an exciting but also somewhat disappointing step. Exciting because wow, Linux on RISC-V hardware in my house that wasn’t crazy expensive, but disappointing as I found the performance to be somewhere between the Raspberry Pi Zero and the Raspberry Pi Zero 2. Ultimately, I didn’t do a lot with it.

Next up, is the much more exciting StarFive VisionFive 2. With super early bird discount, I ordered the 8GB version for less than $100 and was able to get Debian up and running. Now, this is still very early days, requiring firmware updates, custom linux kernels and the use of a USB to TTL console cable (Raspberry Pi directions here: https://learn.adafruit.com/adafruits-raspberry-pi-lesson-5-using-a-console-cable).

But…it was up and running, and the performance was much better, clocking in somewhere between a Raspberry Pi 3 and a Raspberry Pi 4. If we think about this, the industry, in a matter of less than a year, brought Linux RISC-V performance up basically 1.5 generations of Raspberry technology. Looking to the future, in a few days from my writing of this we are expecting Sipeed to allow orders of the Lichee Pi 4A, which promises performance slightly above the Raspberry Pi 4, still within a year timeframe from the LicheeRV.

Embedded programming

So far, I’ve only briefly mentioned embedded. Well, based on my excitement of RISC-V and my close following of Sipeed in particular, I saw an announcement of the ability to run Linux on a tiny MCU. The Sipeed M0S is the successor to the announcement above, is also able to run Linux, and is really, really tiny. Here’s a couple photos, first of the chip:

Then of the chip attached to the dock for development:

Keep in mind…this thing was $4, and can run Linux! Not very well I’m sure, but it’s possible. So, I wanted to play around with it, get down to metal, because considering it is the size of my fingernail, this thing is a beast.

This has sent me (back) into the world of super-low level programming, which I had done a long time ago before the Internet really took off. So far, I’m having fun rediscovering this world, but been running into a lot of “hey, we assume you already know ”, or documentation not in my native language, so I’m documenting my journey here.

Getting started

Pulling this thing out of the box, we have some IO pins, chip, and power/data in the form of a USB-C port. Cool…let’s plug it in. I’m running Linux on my desktop, so I issue sudo dmesg -w when I plug it in to watch what the system does with this thing. I’m presented with:

[8691832.938863] usb 1-10.3: new high-speed USB device number 54 using xhci_hcd
[8691833.063072] usb 1-10.3: config 1 interface 0 altsetting 0 endpoint 0x83 has an invalid bInterval 0, changing to 7
[8691833.063255] usb 1-10.3: New USB device found, idVendor=349b, idProduct=6160, bcdDevice= 2.00
[8691833.063257] usb 1-10.3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[8691833.063258] usb 1-10.3: Product: Bouffalo CDC DEMO
[8691833.063259] usb 1-10.3: Manufacturer: Bouffalo
[8691833.066752] cdc_acm 1-10.3:1.0: ttyACM0: USB ACM device

ok, cool. Linux seems happy with this. What the heck is a USB ACM device? I’m used to something like /dev/ttyUSB0 or something, but not this. A couple searches later, I find this means “Abstract Control Model”, and was used for communication devices back in the day, but is common for MCU programming in 2023. More information can be found here: https://rfc1149.net/blog/2013/03/05/what-is-the-difference-between-devttyusbx-and-devttyacmx/

Now I need to figure out how to flash it, so I pull the BouffaloLab SDK. Later, I found that there is a fork of this repo specific to the M0S. So, we’ll put investigation of the differences here on the to do list, but at a quick glance it looks like these two repos have their first common ancestor on March 9th in 9b5177d.

ok, next to build and run. This…gets tricky as I get a lot of documentation in a language different from my native language, and I’m also not used to this low level work. I keep seeing references to binary blobs checked into repos, and it looks like this is mainly a complaint focused on the actual flashing software. But let’s get the code built first.

For Linux, the environment setup refers to using this pre-built toolchain. I am not a huge fan of this, but we’ll go with it for now and add another item on our to do list to revisit later. I sure as heck am not going to just run this directly on my host, even though it theoretically is built from source much like this. Ultimately I would love to just build with Zig and take advantage of the cross platform capabilities, but again, let’s just get something running.

Docker is my typical tool of choice here, though when I run docker I always run my docker with podman. Daemonless is nice, but it’s also using uid mapping, so I can just run as root inside the container, and files I touch as root there are actually my usual user on the host. It helps with a lot of permissions shenanigans. So with the SDK and the toolchain cloned in sibling directories, and with my current directory the SDK, I startup a container:

podman run --rm -it -v ${PWD}:/build -v ${PWD}/../toolchain_gcc_t-head_linux/:/toolchain --device /dev/ttyACM0 debian:bullseye

Note I’ve passed my device over to the container, although ultimately, I won’t end up flashing from within the container. I’m using debian:bullseye because I’m mostly familiar with debian, that’s what I run on my host, and most SDKs assume you use Ubuntu, which is based on debian and usually close enough for things to work. If you’re following along at home, note the use of --rm means the container will disappear as soon as you leave it, so be careful.

Eventually, I find the following commands within the container work wonderfully:

apt update && apt install -y build-essential
export PATH=/toolchain/bin/:$PATH
cd build/examples/helloworld/
make CHIP=bl616 BOARD=bl616dk
make flash CHIP=bl616 COMX=/dev/ttyACM0

The whole COMX thing tripped me up a bit. First, it’s Windows terminology. Second, what’s with the X? The make commands invoke CMake, and when I screwed up the make commands I often had to delete the build directory inside the helloworld directory.

We’re built! Now to get this the code onto the device. After some trial and error, I find the following procedure to work:

Hold down the boot button
Plug into USB
Release the boot button
Run make flash CHIP=bl616 COMX=/dev/ttyACM0 from the host while inside examples/helloworld in the SDK

After doing this, you can unplug/plug in the device, and magically it’ll work!

Umm..no, of course not. The first time I did this, I noticed that after plugging it back in, my /dev/ttyACM0 is totally gone. My theory is when the device is shipped, it’s shipped with firmware that allows you to upload firmware without pressing the boot button on power on. To get /dev/ttyACM0 back, you need to hold down the boot button while plugging it in. Then you can release the button.

I believe that the boot button simply toggles which firmware is run at startup. With the boot button we get the “firmware loading firmware”, and without, we get “whatever I compiled and uploaded” firmware to run. The helloworld firmware just prints out “hello world” somewhere. It’s our job to find out where. And Linux isn’t showing anything. Being impatient, I don’t have time for any of this detective work at the moment. So, to do list item #3, and let’s see if we can do the universal Internet of Things version of hello world, otherwise known as “get to blinky”.

So, a few short searches later, I stumble on this gist (thanks some random Internet person!). So, I give a quick glance at the code, it looks reasonable, and I build/flash it to the device. Success! I now have two LEDs built into the dev board alternating every 500ms. Current todo list:

What’s the difference between Bouffalo Labs’ SDK and Sipeed’s repo?
Avoid binary toolchains
Figure out what’s up with hello world

We’ll take these in order, so first up:

Figure out what’s up with hello world

Hello world is going to be outputting somewhere. That’s kind of it’s entire purpose in life, after all. There are two reasonable places (maybe a third?) it could output, and my assumption going into the world of “hello world” was that it would be the first. However, that’s clearly not correct:

/dev/ttyACM0
The serial console
(??) JTag. I don’t know too much about JTag, so if it’s that, we have another item for the todo list

If you recall, on flashing the example, my ACM device disappeared. So, we could be at the serial console. Where is that exactly? Time for some digging. We’ll need the following for our detective work.

Because the hello world example is using a bunch of logging statements, I do have some concerns it might be outputting to JTag. So let’s find a different example that’s definitely not using that. Shell looks fun! So we’ll go into examples/shell and see what’s there. There’s an OS and a non-OS version, but we want to remove abstractions and learn this ground up, so we’ll go with non-OS.

Let’s flash it for fun and see what we get. Well, nothing. It’s the same as hello world. No device in /dev for screen or anything to attach to. But this one is much more likely to be intended for use on a serial console. Looking at the source code, we find it’s clearly trying to use uart0:

int main(void)
{
    int ch;
    board_init();
    uart0 = bflb_device_get_by_name("uart0");

Cool. But we need to know where uart0 physically is so we can connect to it. Right now, I assume it’ll be one of the I/O pins we see hanging off the end of the dock. grep -r board_init from the base of the SDK brings up too much noise, but doing another grep -r uart0 yields a few interesting bits. Specifically, There is a drivers/lhal/config/bl616/device_table.c file that looks interesting. Looking at that file ultimately doesn’t give me too much…but let’s put a pin in that one because it may come in handy later. What looks really interesting is bsp/board/bl616dk/board.c, so let’s take a peek there. A few searches through that file looking for uart0, and pay dirt:

static void console_init()
{
    struct bflb_device_s *gpio;

    gpio = bflb_device_get_by_name("gpio");
    bflb_gpio_uart_init(gpio, GPIO_PIN_21, GPIO_UART_FUNC_UART0_TX);
    bflb_gpio_uart_init(gpio, GPIO_PIN_22, GPIO_UART_FUNC_UART0_RX);

    struct bflb_uart_config_s cfg;
    cfg.baudrate = 2000000;
    cfg.data_bits = UART_DATA_BITS_8;
    cfg.stop_bits = UART_STOP_BITS_1;
    cfg.parity = UART_PARITY_NONE;
    cfg.flow_ctrl = 0;
    cfg.tx_fifo_threshold = 7;
    cfg.rx_fifo_threshold = 7;

    uart0 = bflb_device_get_by_name("uart0");

    bflb_uart_init(uart0, &cfg);
    bflb_uart_set_console(uart0);
}

ok - so UART transmit is on GPIO pin 21, and receive is on pin 22. We also have uart speed, parity, data bits, flow control. Everything we could ever want. But where is GPIO pin 21 and 22? Let’s take a look at some schematics. There are two sets of schematics. Ultimately we’re looking for where to plug a TTL to USB device into the DOCK, so we’ll want to look at the dock schematic from the link above. Doing so, we can find GPIO21 and GPIO22 sitting in the bottom right of the schematic where it pictures the BL616 module.

Next, we have to find out where RX0 and TX0 go. Looks like that…is right here:

So we’re looking for J4. It’s not clear from the schematic or the data sheet where J4 is, but J4 has TX, RX and a Ground, so three holes. The board is pretty tiny, so it’s not a lot of real estate to search. The answer seems to be “right next to the USB plug”. I don’t really want to pull out a soldering iron in this process, so I have three questions at this point:

Can I just stick wires into these holes in the board?
Can I move to other GPIO pins?
Can I route communications directly through the USB port I’m already plugged into?

The third option seems especially relevant given I started this journey with a /dev/ttyACM0 device. 2,617 words later though, let’s visit this in another post.

Gitea actions and build badges

Sun, 26 Feb 2023 14:59:00 -0800

UPDATE: With Gitea 1.22.0, build badge support is now built in

Gitea actions and build badges

Gitea actions, at the time of this writing, are new. A feature preview blog post was released in December 2022, and as I write this, Gitea 1.19 RC0 was only published a few days ago in late February 2023.

There is much work to be done to make the feature complete, but I am excited to adopt it as quickly as possible. I have been interested in simplifying my self-hosted setup and removing Drone.io due to the licensing weirdness. I also never automated the reconnection needed between drone and Gitea after a host server reboot/power outage/docker upgrade. Also, I have been impressed with the success of GitHub Actions and its adoption in other systems like Amazon CodeCatalyst. Gitea is the third system I’ve used/been involved with that has embraced the actions format, so it has some industry momentum.

Feature state of Gitea actions

Gitea actions, as it says in the blog post, is in a fairly preliminary state. Here is a short list of what is NOT in Gitea 1.19 RC0, nor do I believe these will be in 1.19:

Cron jobs: PR
An API
Build badges
Artifacts
Services
Documentation (for the most part, you can follow GitHub actions documentation though)
Pull request support

More details are available on the master issue.

Gitea is also riding a fine line of “this is gitea, not github”, and “we want to allow users to copy over their github actions workflows”. Some differences to be aware of:

Gitea actions are in the folder .gitea/workflows rather than .github/workflows
Gitea actions uses property looks at gitea.com, not github.com. Gitea has copied over all GitHub owned actions, but GitHub marketplace actions won’t work without fully specifying the URL
Based on the two above, you might assume that everything has been renamed. But for compatibility, that’s not true. So in your workflow yaml, you still reference ${{ github.<property> }}. I suspect they’ll probably eventually allow use of either GitHub or gitea, but that’s just me gazing into a crystal ball.

Secrets, happily, work just great, and all the other pieces are there, so for a simple, “I’m not going to push this thing too hard”, it works.

Bugs and other weirdness

Gitea actions build runner is still pretty new. It doesn’t support running as a container, but I don’t seem to have a problem running it in a container as long as I mount the docker socket.

However, I did find that trying to use ${{ github.server_url }} in my workflow files resulted in an empty string. I can see the code that sets that property, and my configuration is set properly as far as I can tell, so I’m not sure if this is a bug or a user error. The environment variable $GITHUB_SERVER_URL is set though, so I’ve just used that.

So for a minimal setup, keep the above in mind, don’t push the boundaries too far, and all is good. But…I really wanted build badge support. Coming from drone.io this is a feature I had. GitHub has it. Pretty much every system has it. I’m sure Gitea will eventually include it. But right now, no such thing exists. So, time to code a workaround.

Getting build badge support

A couple things, before getting started.

I’m looking to enable build badge support for my public repos. I don’t really care about private repos
I am confident that this will be built in to gitea proper, so a hacky short-term workaround is fine.
There’s no API support
I don’t want anything elaborate

So with that in mind, I’ve turned to the following services to build something in a couple hours:

Shields.io to provide the imagery
Cloudflare workers and Cloudflare generally for DNS and proxy support

My plan was relatively simple:

Setup a custom subdomain actions-status.lerch.org that has all requests handled by Cloudflare workers
Use a worker to probe the gitea interface, scrape the HTML (YUCK! But there’s no API…), and determine the latest build status
Build the appropriate URL to shields.io and fetch the output
Return the output to the caller

Two hours later, I came up with this solution, which works pretty well. One thing I noticed it doesn’t do is differentiate between a canceled workflow and a failed one. It turns out, the HTML is identical on the summary screen, so another call would be needed to determine “canceled”, and covering that edge case didn’t seem all that necessary.

If you’re interested in a similar solution, make sure you change git.lerch.org below. Also, the badge styling may not be your cup of tea so to speak, so you may want to update the lines where the badgeUrl variable is set starting at line 44.

I kind of like using the gitea logo on the badge. I feel kind of happy that I’m likely the first person to use a gitea badge build based on a gitea actions run on a project. You can see the badge in action on my aws-sdk-for-zig project page or it’s GitHub mirror.

NOTE This was done with Gitea 1.19 RC0. Because HTML returned is so fragile, it’s ~~possible~~probable that the web page scraping breaks with other versions. With any luck, an API will be available before that happens, but who knows. To save you reading the code, the technique used is to visit the URL https://git.lerch.org/<owner>/<repo>/actions?state=closed[&workflow=<workflow>.yaml]. When the html is returned, the code finds the first instance of an element with a commit-status CSS class, then captures whether it should be red or green (these are the names of two other CSS classes used). Green is success, red is either failed or canceled. All this fragility is in the parseResponse function starting line 19 below.

export default {
  async fetch(request, env) {
    const getOrigin = function(components) {
        console.log(components);
        const owner = components[1];
        const repo = components[2];
        let workflow = components.length >= 4 ? components[3] : "";
        if (workflow && !workflow.endsWith('.yaml')){
          workflow = workflow + '.yaml';
        }
        console.log('owner: ' + owner);
        console.log('repo: ' + repo);
        console.log('workflow: ' + workflow);
        console.log('workflow length: ' + workflow.length);
        const query = '?state=closed' + (workflow ? `&workflow=${workflow}` : '');
        return `https://git.lerch.org/${owner}/${repo}/actions${query}`;
    }

    const parseResponse = function(text) {
      const regexp = /commit-status.*icon (green|red)/;
      const match = text.match(regexp);
      console.log(match);
      if (!match) { return null };
      const textStatus = match[1];
      if (textStatus === 'green') return 'succeeded';
      if (textStatus === 'red') return 'failed';
      console.log('unexpected action icon status: ' + textStatus);
      return null;
    }

    try {
      const { pathname } = new URL(request.url);
      const components = pathname.split("/");
      if (components.length < 3 || !components[2]){
        return new Response("usage: /<owner>/<repo>[/workflow]\nNOTE: workflow does not require .yaml extension", {status: 404});
      }
      const origin = getOrigin(components);
      console.log(origin);
      const originResponse = await fetch(origin);
      if (originResponse.status != 200){
        return originResponse;
      }
      const status = parseResponse(await originResponse.text());
      let badgeUrl = 'https://img.shields.io/badge/build-';
      if (!status) {
        badgeUrl += 'unknown-blueviolet'
      }
      badgeUrl += status + '-';
      badgeUrl += (status === 'succeeded' ? 'success' : 'red');
      badgeUrl += '?logo=gitea';
      console.log(badgeUrl);
      return fetch(badgeUrl);
    } catch(e) {
      return new Response(e.stack, { status: 500 })
    }
  }

}

First thoughts on Zig

Mon, 18 Jan 2021 12:23:18 -0700

First thoughts on Zig

I encountered Zig a while ago and it intrigued me. It’s intrigued me enough to write a small utility program for personal use and now an SDK for AWS, with some contributions to upstream projects along the way. All told I’ve written more than 10k lines of code in the language (current net code count is just shy, but I’ve done a lot of refactoring!). My current opinion is that zig, for me is a great blend of high level/low level with a focus on efficiency. My intent at this time is to use zig whenever possible.

If I can try to summarize the goals of the language, it would be this:

Provide a systems programming programming language that:

Is better defined than C
Has first class support for cross-compilation (both OS and processor types)
Provides a high degree of safety
Can be used for embedded (bare metal/no OS) development

Provide a general purpose programming language that:

Doesn’t need garbage collection
Is simple
Generates small/fast executables

When working in the language it sometimes feels like I’m programming in go and sometimes in c. I like the “simplicity” of the language, but more on that later.

Should you use it? Probably not…but that’s due to its one fatal flaw right now, which is that it’s just not baked yet. At the time of this writing, it is at version 0.8.0 and sees hundreds of commits per week. Some of these are breaking changes, and depending on your needs, some features may be missing or broken. Documentation of the standard library is a lot of “go read the source code”. Parts of the language and library don’t follow naming conventions or idioms that are still evolving and changing in the language.

Comparison to other languages

I mentioned above it sometimes feels like go and sometimes like c. I think spiritually these are zig’s closest languages, and I’ve heard zig described as one of the languages that are trying to be “a better C”. If we remove garbage collection from go, we get to something pretty close to zig. So, I think these two languages are a good place to start:

Like go, zig is a simple language. Its grammar is currently 554 lines. If you understand go, you can get to zig (mostly) by:

Removing closures
Removing garbage collection
Removing goroutines
Removing tabs ;-)
Adding spaces
Adding a sane error handling mechanism
Adding “generics” (more later)
Adding first class embedded development

For loops in zig reminds me of the range-for loop in go. However, zig uses the same capture syntax from for loops in its while loops, if statements and switches, which gives a nice consistency to the language. Declaring public things in zig is explicit, unlike the capitalization-based concept of go. Personally I like the explicit way of doing things, but I see the rationale of go here. zig also disallows tabs in the source code, which is also my jam, but it certainly bothers some.

Also worth noting is binary size. As of go 1.15, a hello world, even without CGO Enabled, is just shy of 2MB. there is an epic issue talking about this. zig’s hello world, built in default debug mode is 570k. Switching to release mode will give you a 69k binary that when stripped will be approximately 2k. While memory utilization is of course variable, I’ve directly converted a go program to zig and easily seen a 10x improvement on that front. When you have to allocate/deallocate memory yourself, you’ll be a lot more careful. A notable advantage of go over zig is that the compiler is super-fast in comparison.

Of note, since zig is pretty new, it’s important to be able to interface with C libraries. Zig makes this more or less seamless, and since zig can be used as a drop in replacement for cc, this can be done with the same zig download you use to compile zig code. In my SDK, I had to work around zig’s lack of direct bitfield support. The solution here was to simply drop a C file in the middle of my project and have zig compile it with everything else. Using C from Go feels like a bolt-on compared to zig, and is much harder.

Like C, zig is focused on portability across operating systems and hardware. In 0.8.0, it has Tier 1 support for 8 architectures and 4 Operating Systems in addition to freestanding/ bare metal development. Tier 2 support adds 3 more Operating Systems plus UEFI, and 3 additional hardware architectures. However, C leaves a lot of underspecified behavior, which makes a lot of compiler differences. Lack of C bitfield support in zig is pretty much blocked due to compiler differences stemming from underspecified behavior in the C language. Of course, complaining about compiler differences may not be entirely fair as zig only has a single compiler implementation, but the expectation of a language should be that the specification prevents a lot of undefined behavior. In zig, Nulls, must be explicitly allowed, and will be checked. Undefined variables are possible, but also checked by the compiler to be defined before use. Zig also adds defer and errdefer, which provide the ability to handle a lot of memory (or other resource) deallocation edge cases. As zig matures, additional checks are being investigated and added. Most recently, after 0.8.0 a change was made to detect and prevent unused variables.

Rust

Rust tries to find “zero-cost abstractions” to provide memory safety in an object oriented environments. I think of the language as C++, but with memory safety and without a lot of cruft that C++ started with and has gotten worse over time. Like zig, cross-compilation of Rust is easy. Like Go, incorporation of C libraries requires more work than zig. Because Rust has a first-class concept of memory “ownership”, allowing Rust to control allocation and de-allocation of memory. In zig, all memory allocation and deallocation is explicit. Zig’s approach can be painful at times compared to Rust, but this pain also forces the programmer to consider allocations carefully and in practice probably reduces the working set of many programs. It also means that segfaults are part of life, as are memory leaks, although zig test and choosing the right allocator can help debug leaks. Rust is much more complicated than zig, and compilations are much slower. The toolchain is also much heavier, with image on-disk size of 806MB for docker 1.53-alpine version. Zig’s toolchain, uncompressed on disk weighs in around 300MB by comparison.

Other

Sorry, other languages, you get bucketed here. Java and C# can join most of the go discussion, except that these are true object oriented languages, while zig is procedural. Functional languages have great attributes and I like them a lot, but they are typically divorced from the hardware they are running on, while with zig this is decidedly not the case.

memory

Memory management, without a garbage collector, is clearly a focus of the language. Allocations and deallocations are explicit just as in C. However, zig has an approach that works well for managing deallocations to prevent a lot of issues. In zig, you can return an error, which feels a little bit like exceptions in languages like C# or Java. In fact, zig has a keyword try that will automatically return an error if an error occurs within the expression, and a keyword catch that will allow you to “handle” an error. To avoid missing a deallocation for exceptional situations, you can use defer statements for deallocation (or tear down for other resources). For situations where returned memory will be owned by the caller, another keyword errdefer can be used. This statement basically says, “If an error happens in this function, deallocate/deinitialize, otherwise the caller is responsible”. In practice, this works really well once you start getting the hang of it. I have, however, had my share of segfaults and memory leaks in testing before understanding how to build tests and structure allocators to detect leaks.

safety

Zig has an ongoing effort to improve safety, so this section will change a lot as the language evolves. Right now, there are built in test facilities around memory leaks. There are checks for buffer overruns and integer overflows. There are pointer alignment checks, asynchronous checks, and array bounds checks. Much more is planned, and the project culture is to eliminate as many “foot guns” as possible in most scenarios. It is possible to turn checks off in lieu of performance and executable size, though that isn’t recommended. Null values must be explicit, as are undefined variables.

async

Async operations are designed to be transparent. Unlike many languages, a function is free to do its own thing, then suddenly find itself executed asynchonously. There are some subtle differences between zig, and say, Rust here in terms of implementation as well, but the philosophy in zig is to allow a lot of code reuse, and the async approach enables that, as does its comptime behavior, as I will talk about next.

comptime (and generics)

Comptime is the most unique part of the language IMHO. Functions can run at runtime or at compile time (comptime) “without knowing the context”. These quotes are pretty important, because in my experience, compile time code is super powerful, but as a programmer you absolutely have to consider both compile and runtime behavior of functions as well as the partial compilation of functions that result in part compile, part runtime behavior. Used well, the ability to run arbitrary code at compile time is an incredibly powerful tool. But understanding how this works and how to take advantage of it is easily the biggest learning curve of the language. The success of the language, in my opinion, hinges primarily on two factors. First, zig might be too late. As an industry, Rust might be “good enough”, and zig may not get traction as a result. Secondly, comptime semantics might be too hard to reason about, or alternatively, might be the killer feature the industry has been waiting for.

As an example, you may be coding away and ultimately write a function that starts looping through the fields of an enum because you’re doing some light metaprogramming. My recent example here is using enum values as switches to a command line program. Anything metaprogramming becomes comptime. First, you’ll get an error you probably don’t understand. The solution is to use an inline for, which you’ll eventually figure out, but understanding the compiler behavior may be beyond some new programmers. This is complicated by the fact that throwing in some “printf debugging” won’t do you any good because that, of course, is runtime, so you need to use @compileLog statements instead, which can look really weird when doing a zig build command.

The other aspect of this that concerns me is that the system is so powerful that it may make IDEs unable to assist the programmer. For example, zig has anytype, which is kind of equivalent to var in many languages and is usually used for function parameters. It basically says, “I don’t want to worry about the type of this thing right now - I know what I’m doing”. anytype will get replaced with the actual type at compile time, so you don’t lose safety, but in order for an IDE or language server to provide completion information, the code needs to be fully evaluated, including all call sites. As a practical matter, I don’t believe I’ve seen any intelligent completion based on anytype parameters, only my brain saying “I know it’s anytype but the thing I get will have a function foo() on it”. To be fair, my brain needs to handle this with dynamically typed languages like JavaScript or Python, but zig is typed and it feels inconsistent here.

The nice thing about comptime, though, is the power it provides (or promises). There are still gaps here including big ones like the inabillity to allocate memory at comptime. These plan to be fixed, but even without memory allocation amazing things are currently possible, like the ability to parse json data at compile time. This pushes a lot more computation up front and makes executables much faster as a result. It also allows generics in what is otherwise a “generics-free” language, because the return type for a function, for instance, can be calculated by another function that runs at comptime! It’s amazing, and I’ve already overused this several times before ultimately backing down to go a more pedestrian path.

“feel”

Let’s get some of the bad out of the way first. Error messages…omg. To zig’s credit, it does a fantastic job providing the detail you sometimes need. But let’s look at an example I hit all the time. This line of code prints as you would expect:

std.log.info("All your codebase are belong to us.", .{});

If I wanted to print a value, I could say:

std.log.info("All your codebase are belong to us. foo={s}", .{foo});

But if I forget to put in the {s}, like this:

std.log.info("All your codebase are belong to us. foo=", .{foo});

I will get a screenful of the error return trace including all 12 calls that were in the chain, even if this is only in the main function. You get used to it, but it leaves you with a certain sense of dread seeing these types of errors.

That’s cosmetic, but one thing that’s a bit more tangible is something I miss from a few other languages, most notably Rust. That is, when coding I sometimes really just want everything to be an expression. Zig has an example of returning from a block, for instance, that just feels a bit wonky. This example is in the current documentation for example:

// use compile-time code to initialize an array
var fancy_array = init: {
    var initial_value: [10]Point = undefined;
    for (initial_value) |*pt, i| {
        pt.* = Point{
            .x = @intCast(i32, i),
            .y = @intCast(i32, i) * 2,
        };
    }
    break :init initial_value;
};

While I’m glad this is possible, Rust would handle that by just ending here with a last line of initial_value (excluding a semicolon), indicating that the block’s “value” is initial_value. This feels a bit more natural to me, though I’m sure there are lots of reasons why other parts of zig make this undesirable or impossible. I’m not a language designer, but I do get to write words on the Internet. ;-) I will note that shortly after writing the initial draft of this post there was a short discussion by the zig team on IRC touching on this in particular as being a bit awkward.

The focus of the language is currently on building a self-hosted compiler, so it’s not really fair to pick on other things quite yet, but I’ll take a quick shot anyway. The standard library really needs some attention. It is inconsistent, even with regards to naming standards, and is poorly documented. The idioms used are also inconsistent, which I believe leads to confusion. I hope that the core team can get some focus time on it soon.

Which brings me to one of the bright spots, actually, which is the community. I’ve lurked on IRC, seen the Issue and PR traffic, and seen the twitter threads. Andrew and it seems everyone involved with the language is passionate and helpful to anyone regardless of their knowledge of zig. I’ve yet to see a hostile attitude from any zig contributor.

Multi-architecture docker builds: notes on ARM

Thu, 28 May 2020 16:16:48 -0700

Multi-architecture docker builds: notes on ARM

There are significant dragons in trying to support a wide variety of ARM chips. There are also a significant number of dragons in multi-architecture Docker builds. This post gathers some of my experiences working with multi-architecture docker builds.

ARM has emerged as an important processor architecture. Combining the ARM Architecture/ISA, for which a license can be purchased, with custom silicon, several companies have had a lot of success in the market:

Low cost system on a chip (SOC), like Raspberry Pi devices
Android and iPhone devices
AWS, through custom Graviton chips
Apple M1 Macs

My hope is that another up and coming processor, RISC-V, gets equivalent market penetration. The advantages of this ISA basically comes down to the fact that it is a RISC architecture, consistent and open source. But ARM is a great step as many companies can (and have) licensed ARM architectures, and as such multiple competing implementations exist.

Linux processor architectures:

Using uname -m will provide the CPU architecture on POSIX systems.

On my Raspberry Pi Zero, we see something like this:

$ uname -m
armv6l

We can now translate this to:

Arm: Obviously, this is ARM v6: version 6. Generally version 5, 6, 7 and 8 are still supported in 2021. l: little endian, which is fairly common

In ARM-land, the issue of whether or not the CPU has a hardware floating point unit is a big deal. Generally processors that you see will be “hf” in 2020, but you cannot count on it, especially in low end or embedded devices.

For ARM 64 bit architectures, this command will simply provide the output “aarch64”.

I believe mostly uname -m for linux would found via the following search across its source code: https://github.com/torvalds/linux/search?p=1&q=UTS_MACHINE. This would surface all the definitions for the variable UTS_MACHINE, which is in turn pulled when running uname -m. ARM being a little different, ARM 32 bit is a calculated value that doesn’t show up in these results. Take a look at https://github.com/torvalds/linux/blob/95f05058b2bbe3b85c8617b961879e52f692caa5/arch/arm/Makefile for a reasonable picture of the 32 bit landscape for ARM.

Debian architecture names

Debian (and derivatives, like Ubuntu), being such a huge portion of the linux landscape, we need to consider their architectures. These are defined via their ports. Most relevant ARM ports are officially supported. Here is the list, descriptions taken directly from the page:

armel: The oldest of the current Debian ARM ports supports little-endian ARM CPUs compatible with the v5te instruction set.
armhf: A lot of modern 32-bit ARM boards and devices ship with a floating-point unit (FPU), but the Debian armel port doesn’t take much advantage of it. The armhf port was started to improve this situation and also take advantage of other features of newer ARM CPUs. The Debian armhf port requires at least an ARMv7 CPU with Thumb-2 and VFPv3-D16 floating point support.
arm64: Version 8 of the ARM architecture included AArch64, a new 64-bit instruction set. Since Debian 8.0, the arm64 port has been included in Debian to support this new instruction set on processors such as the Applied Micro X-Gene, AMD Seattle and Cavium ThunderX.

In Debian-land, armhf= armv7 32 bit

Unofficially, there is also:

arm: This port runs on a variety of embedded hardware, like routers or NAS devices. The arm port was first released with Debian 2.2, and was supported up to and including Debian 5.0, where it was replaced with armel.
armeb: ARM, software Emulated floating point, Big endian. I have seen references to this, but not seen any in the wild.

32 bit architectures

These are important when building for compatibility across a wide spectrum of devices:

arm5: I believe this is mostly obsolete
arm6: Used on Raspberry Pi Zero/Zero W and some low end and/or old android phones Few docker containers support this architecture
arm7: Considered “hard float” by most, and seems to be the most popular in the wild
arm8: This can work for 32 or 64 bit, but generally 64 bit is used for v8-specific builds

In docker, these are all considered “arm” architecture. Docker has introduced a concept called “variant”. In docker hub, you will see arm7, for instance, noted as “linux/arm/armv7” In docker hub, you will see arm7, for instance, noted as “linux/arm/v7”. This last portion is the variant.

64 bit architectures

Arm64/Aarch64 seems a lot more consistent. Since 2011, there has been only arm8.x, and software compiled for arm8/arm64/aarch64 only seems to need one variant. Arm64/aarch64 is used depending on who you are

Raspbian and Debian disagree on what armhf mean. Basically Raspbian says ‘armhf’ includes any arm processor with hard float. Debian uses armhf to mean arm7+ 32 bit architectures. However, arm8+ are 64 bit, so armhf in Debian realistically means just

readelf -A /proc/self/exe | grep Tag_ABI_VFP_args && echo 'Hard floating point' If this command outputs a line “Hard floating point”, Raspian will use armhf packages. However, debian requires arm7 to use armhf packages.

Docker is a mess because arm variant detection basically does not exist

Building an image

Sticking to the Docker ecosystem, I’ve had some success with multi-architecture docker builds generally following the advice on this page: https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408

Gotchas I’ve run into:

Everything with multi-architecture images is experimental. You need a recent operating system, a recent docker, a recent kernel, and you need to enable experimental features. There is even a table on the post above that walks through this. If you’re host environment doesn’t match a working configuration from the table in the blog post, then go back and fix that first, or you will be frustrated. Follow this carefully - even on a pretty modern system (debian buster new install) I needed to run their “fix-it script”, and even that script needed some tweaks.
Once you have qemu-static and all the configuration, it’s pretty seamless to run programs compiled for another architecture. It can get super confusing as everything “works”, but then on deploy, it doesn’t, because qemu stepped in during your testing and you didn’t even realize it. file is your friend, docker pull is not.
docker pull --architecture works as you expect, but if you, for instance, do docker pull --architecture linux/aarch64 myimage:latest, then later do docker pull myimage:latest expecting amd64 (for Intel or AMD CPUs), you will not get what you expect. Docker will happily look at its cache and say “yeah, I’ve already got that image” and serve up the wrong CPU architecture. And you won’t notice until much later when everything is broken and you’re stuck in the office while your team is out drinking (best case) or hovering over your shoulder (worst).

This post is also useful: https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/ However, it also focuses on buildx, which is a docker cli plugin (another piece of software to install, and only enabled with DOCKER_CLI_EXPERIMENTAL=enabled.

Because I like podman due to its lack of daemon and rootless containers, I’ve taken a shot at making this work there as well. The good news is that the manifest commands are similar (though not identical). The bad news is that pulling different architecture images just seems like a broken situation. I’m sure it can be fixed, but my current drone install is on docker proper, so I haven’t had a chance to investigate yet.

PKCS 11, OpenPGP, Yubikeys/Solokeys, and Windows AMIs

Mon, 02 Dec 2019 16:16:48 -0700

Using single key for both PKCS#11 (PIV app on Yubikey) and OpenPGP/GnuPG

I was looking at creating a Windows instance on AWS EC2 over the weekend, and I started thinking about the administrator password. In AWS on Linux and likely other Unix-like OS’s on EC2, you can provide a public SSH key and through the magic of cloud-init, the public key is placed in the .ssh directory of the user, which varies based on the AMI chosen.

Windows on EC2, however, follows a different process. Here, the SSH public key is used a generic public key. When the Administrator process is set by EC2Launch, That password is encrypted using the public key provided, then you can decrypt the data with your private key. This can be done through the console, or with the command line aws ec2 get-password-data. The console, however, requires that you upload your private key, while the CLI version only encourages it.

In my setup, pretty much the only key pair I use is generated via gpg on an airgapped machine running Tails. The private key is copied to two USB sticks (primary and backup), then imported onto a Yubikey NEO, which is still somewhat open source (Sorry, Yubico, but I respectfully disagree). I’m pretty happy with this arrangement, but it means that I can’t provide my private ssh key to an API or a console (nor would I want to).

I started thinking about how retrieval of this data could work in an environment with a Yubikey or an HSM. GPG is strict about the data it decrypts - it must be in the PGP format, which is not how get-password-data provides it’s data. Generally, the idea here is that if the private key is not provided, the data will be returned encrypted and base 64 encoded. The idea then is to base 64 decode the data, write a file out, and issue a command like openssl rsautil -decrypt -inkey mykey <mypassword.bin to decrypt it. That command may not work - it’s just an example.

Without direct access to the private key, using a command such as the above is not possible. Nor is the use of GPG. The solution, then, is to use the Yubikey’s support for PKCS#11. This is provided by the PIV applet on the key. This applet, however, has completely separate storage from the OpenPGP applet, so the keys, PINs, etc are all managed separately, even if the concepts are shared between OpenPGP (accessed by GPG) and PKCS#11 (accessed by PKCS client tools like pkcs11_tool).

In an ideal world, then, what I’d like to have is a single key, loaded into both OpenPGP and PIV applets, used for SSH access (via gpg-agent) and Windows passwords (accessed via pkcs11_tool). If it were just a question of SSH, we could remove GPG entirely, but I also use the GPG key for commit signing, so I’m in a GPG world anyway. The GPG key has separate subkeys for signing, authentication and encryption, so that’s good separation of duties. It’s worth considering that I’m breaking this separation with the procedure below and it might be best to have two EC2 KeyPairs, one for Windows and one for SSH, but the differences in the procedure amount to the selection of a different subkey, and the rest is just discipline. GPG does the enforcement, and we’re explicitly taking out that enforcement.

Generally, what I was looking to do is the following:

Export the gpg private and public key
Convert the gpg private key to PEM format
Load the private key into the PIV applet on the Yubikey
Use PKCS#11 interface to decrypt the password data

Step 0: Getting started

If you’re like me and set up a GPG key with an expiration that needs to periodically be extended, this process will need to repeat. However, this particular step will only need to be done once. For that reason along with the fact that numbering should start at zero, this one time only process will be step 0.

First things last: I had significant confusion when working with Yubikey and PKCS#11. Nothing I did worked. I found that GPG agent (probably) was blocking all PKCS#11 access. Simply removing and reinserting the key was able to unblock me.

So we basically have a default key. Even if it was configured previously, none of that has anything to do with PIV, so forget that. You’ll need the yubico-piv-tool to configure the key. Even though it speaks PKCS#11, the configuration is unique to Yubikey. This gist does a good job describing the steps, however, it’s focused on on-key generated keys, which is a problem if the key is lost. For now, we’ll just get the Yubikey initialized, changing the three management pins to something other than their default values:

  # Set the managment key: must be exactly 48 characters
  yubico-piv-tool -a set-mgm-key

  # Set the device PIN (user pin - default 123456)
  # Since this default is well known, it's ok to pass it on the command line
  yubico-piv-tool -a change-pin -P123456

  # Please set the device PUK (Admin pin - default 12345678)
  # Since this default is well known, it's ok to pass it on the command line
  yubico-piv-tool -a change-puk -P12345678

Step 1: Export the gpg private and public key

This step is easy, but has it’s nuances. Private keys should be handled with extreme care. For me, that means the key is stored on a physically secured USB key (with a secondary backup key), and all key operations are on an airgapped computer running Tails. Since these files are only used for this process, they can be managed from on the tails filesystem, which will be wiped on shutdown.

  # Replace mykey with the uid of your key. This is often your email address.
  # It can be seen in the command gpg --list-keys on one of the uid lines
  # If you have multiple uid lines, any one of them will work
  gpg --export-secret-key mykey > mykey.gpg # OpenPGP format secret key
  gpg --export-ssh-key mykey > mykey.pub    # OpenSSH format public key

At this stage, we’ll have the public key in OpenSSH format documented in RFC 4253. This will work for our needs. However, the GPG private key is in the OpenPGP Message Format for key material and as such needs a conversion to PEM format to be useful.

Step 2: Convert the GPG private key

At this stage, we need another tool. The MonkeySphere project is focused on expanding GPG to other uses, and the packages they provide allow for conversion of keys among other things. So, we can just use the tool they provide to do the conversion. However, there is one caveat: the tool cannot convert password-protected keys. If you’ve been managing keys securely you certainly have a password protected key, so we need to remove this. It is for this reason you really want to do this on an ephemeral file system. We’ll use the gpg --homedir option to override the normal home directory. Choose something that will self-destruct, remove the passwords from the key, do another export, and the conversion can work. This looks like the following:

  # Replacements:
  #
  # * ephemeralDirectory: your ephemeral directory name
  # * mykey.gpg: your exported file from Step 1
  # * mykey: key uid, which will be output during the gpg import command
  #          when it says key <keyid> public key "your uid" imported
  # * DD53AC86: This is the key id from the authentication subkey of your
  #             key. Get this id from the command
  #             gpg --list-keys --with-subkey-fingerprint
  #             Look for the line that says something like
  #             sub rsa2048 <date> [A]
  #             A is for Authentication, and is what is used for SSH
  #             The key id is the **last** 8 characters of the next line
  gpgtemphome=myephemeralDirectory
  mkdir $gpgtemphome
  chmod 700 $gpgtemphome
  cd $gpgtemphome
  gpg --homedir $gpgtemphome --import mykey.gpg # you will be prompted for the password here
  gpg --homedir $gpgtemphome --passwd mykey     # At this point, you will be prompted
                                                # for new passwords, leave them
                                                # blank, then confirm that you want
                                                # a blank password. This will likely
                                                # happen 3 times, once for each
                                                # of the subkeys of the imported key
  gpg --homedir $gpgtemphome --export-secret-key mykey > "mykey.gpg.nopass"
  openpgp2ssh DD53AC86 < "mykey.gpg.nopass" >mykey.pem # pem format secret key

If you were successful, you’ll have a PEM format secret key file in $gpgtemphome/mykey.pem that can be now be loaded onto the Yubikey. The hard part is now done.

Step 3: Load the private key into the PIV applet on the Yubikey

All PKCS#11 operations on the Yubikey work in slot 9a. So, we need to load the key into slot 9a of the PIV applet. It’s almost, but not quite, that simple. For this to operate correctly, we need a self-signed cert rather than just the key. The following commands will import the key, create the cert, and load it. Most of these commands will also require the management key to operate successfully. Anytime you see -k below, you’ll be prompted.

Note that pin-policy and touch-policy parameters only apply to Yubikey 4. I’m working with the (sort of) open source Yubikey Neo. You may want to adjust these to your taste. The -v turns on verbose mode, which I found helpful when the terminal wasn’t pasting the management key.

  # This will prompt for the management key, without which you cannot import the key
  # Replace mykey.pem as appropriate
  yubico-piv-tool -s 9a -a import-key -i mykey.pem -k --pin-policy=once --touch-policy=always -v

  # Generate the the certificate to load. Replace mykey.pub with the file
  # exported in Step 1
  ssh-keygen -e -f mykey.pub -m PKCS8 > mykey.pub.pkcs8

  # Replace mykey.pub.pkcs8 name as appropriate from the command above. This creates a "mykey-cert.pem" file
  # Yubikey piv tool requires the file to be named with a pem suffix, so don't get fancy with the file sufix above
  yubico-piv-tool -a verify -a selfsign-certificate -s 9a -S "/CN=SSH" -i mykey.pub.pkcs8 -o mykey-cert.pem

  # Finally, we can import the certificate we generated into slot 9a.
  # This will ask for the management key
  # Yubikey piv tool requires the file to be named with a pem suffix, so don't get fancy with the file sufix above
  yubico-piv-tool -a verify -a import-certificate -s 9a -i mykey-cert.pem -k

At this point, you should be able to verify the key is properly loaded on the Yubikey. Your gpg exported ssh public key (in my example, “mykey.pub”) should match what comes off the Yubikey via PKCS#11. The ssh key from gpg will have a comment - the command below uses the Unix command cut to strip that out. As usual, replace the mykey.pub name with the name you used:

[ "$(cut -f1-2 -d' ' <mykey.pub)" = "$(ssh-keygen -D /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -e)" ] && \
  echo 'File and smartcard match'

With any luck, you should see the output File and smartcard match. If you don’t, something went wrong. You can also test encrypt/decrypt operations with something like this:

echo 'it worked' | openssl pkeyutl -encrypt -inkey mykey.pem -pubin > encrypted.bin
pkcs11-tool --decrypt -v -l --input-file encrypted.bin  -m RSA-PKCS # Prints 'it worked'

That should execute without errors and print ‘it worked’. From the first test we know that the ssh key is properly loaded in slot 9a, and the second proves that decryption from the Yubikey using PKCS#11 works properly.

We’re done, and you can now delete the files we’ve been generating along the way. In my example, that’s:

mykey.pub
mykey-cert.pem
mykey.pub.pkcs8
mykey.pem
mykey.gpg.nopass
mykey.gpg
encrypted.bin (if you ran the last test)

Assuming you’re doing this on an airgapped computer, go ahead and shut down and remove the Yubikey.

Step 4: Use PKCS#11 interface to decrypt the password data

We’re loaded up. If you’re doing all this for the same reasons as I, go ahead and launch an EC2 Windows instance. You’ll need to establish an EC2 keypair if you don’t already have one, and specify that key pair when you launch the instance.

Once the instance has launched, we’ll use the CLI command get-password-data to retrieve our data. It comes in a JSON format, base64 encoded, so the command will look like the below. Note we need a file as the pkcs11_tool does not read from stdin (as it needs to prompt for the user PIN). Also, the password is not established immediately, so you may need to wait a bit for the data to appear.

aws ec2 get-password-data --instance-id <blah> --query PasswordData --output text| base64 -d > encrypted-adminpass.bin
pkcs11-tool --decrypt -v -l --input-file encrypted-adminpass.bin -m RSA-PKCS

Connecting to the Windows instance.

There may be an issue connecting, as newer Windows AMIs enable NLA by default. Simple Systems Manager has an agent installed by default, so in these cases you can attach an appropriate IAM role and issue the following command, replacing the instance id below with yours:

aws ssm start-automation-execution --document-name "AWSSupport-ManageRDPSettings" --parameters "InstanceId=i-03033520993ddf97f,NLASettingAction=Disable"

But wait, what does all this have to do with Solokeys?!

The Solokey is a truly open source security key. For now, it handles only Fido2 and U2F. However, they’re working on OpenPGP support. PKCS#11 would be over and above, but it’s possible with something like scute that PKCS#11 support could be provided via OpenPGP. In this process, however, I’ve heard rumblings that scute is somewhat unstable. It’s also unclear whether the authentication subkey would be allowed to work for decryption in this particular use case.

More information

The following posts/links helped me immensely on this journey:

Updates

Just after publishing this post, this link came across an email list I subscribe to. In a nutshell, two Python libraries were caught exfiltrating private SSH and GPG keys. Storing your private keys on your Yubikey completely render this attack moot, as no private keys are available in any way.

COULD_NOT_CREATE_SYNC_ACCOUNT Amazon Prime Video, and why root access is important

Wed, 02 Oct 2019 16:16:48 -0700

Early this year, an automatic upgrade to my Amazon Prime Video application was installed. I was happily enjoying Mission Impossible, Season 1, downloaded to the device for use on planes, but the next time I opened the application, I was greeted with “COULD_NOT_CREATE_SYNC_ACCOUNT”, and the Prime Video app refused to load. Well, so be it…I moved on to other, likely more productive, usage of my airplane time. However, this error message bugged me, and a 50-ish minute show is pretty good for taxi+takeoff+climbout to 10k ft (also, 10k feet descend to final approach and landing). So, it stayed in the back of my mind.

Fast forward a couple months later, maybe in the summer, and I contacted Amazon Prime Video support. Amazon support is appropriately customer-obsessed, and the engineer was relatively helpful, but clearly followed a script that I’ve played through before. Of course, I got to the point of “reset your Android device to factory settings”, at which point I pulled my “no way” card. Support did fall down somewhat at this point, where I was promised a follow-up to my issue, but this did not occur.

A couple months later still, this problem still bothered me. For reasons, I actually had access to the source code for the application (actually, library) that produced this error message, but ultimately this wasn’t important, other than to point out the fact that I spent some crazy side time working out how to build the application and all dependent libraries. It was a great learning experience, but ultimately it wasn’t the easy path here.

I started to consider what the error message was trying to tell ~me~ a developer, and it dawned on me that they were talking about the Android accounts feature. I looked at my phone, another Android device, and noticed that indeed, there was an “Amazon Video Sync” account listed in the “Accounts” section of the settings. I had never created this account manually, so the upgrade clearly tried to install this account programmatically and had failed. Ok, now that I can work with…

The account had no settings, but in my further research I found that this is super-useful from Prime Video’s perspective. Android handles all the ugly work of connectivity and retries and exponential fallback with jitter, battery saving, etc., and simply calls into the class when everything is good to go. Then Amazon Prime can sync. It’s totally cool here to download videos and whatnot based on a sync provider. Android Prime Video is just trying to install the sync provider, and cannot. It needs this, ostensibly to download videos, but due to the failure, it cannot proceed and fails. All reasonable, my only question being - should a missing sync provider really fail online viewing? It seems this should be limited to download for offline videos.

ok, so the application can’t install the sync provider. Let’s dive in for why. So, I first tried to install manually. This was interesting. The account settings in Android simply hung when I tried to add the account manually. Well, this is interesting…

Root access required

A few DuckDuckGo searches later, and I find that deep in the depths of Android, two files control the sync providers shown on the screen (and presumably in the APIs of Android). These are, at least on my machine, though I think it varies by Android version:

/data/system_ce/0/accounts_ce.db
/data/system/sync/accounts.xml

So…the two files worked together. Using termux with the sqlite3 package installed, I opened the database. This stuff is a good way to mess up your system, so I’m not providing further instructions on how this is done. A quick select * from accounts yielded 35 accounts, 34 of which were affiliated with Amazon Prime Video, the “name” field varied between ‘Amazon Video Sync n’ and ‘hollywood-sync-account_n’, with the latter being more recent, but also interchanged with a “AmazonSyncAccount-_n’. This was clearly a result of my multiple attempts to fix the problem. I created a backup of the database, ran a delete from accounts where _id > 1 (your id will obviously vary), and the table looked good. I double checked all the other tables in that database, but nothing else looked out of line.

A reboot later and a running of Amazon Prime Video and…no love. Something so obviously wrong had been fixed, but I still had the same error. I checked the database, and sure enough, I had a new entry, but things were still off. So, I deleted again, then checked the accounts.xml file. The last entry in the XML was an account named “Amazon Video Sync”, so I deleted that file in addition to the relevant accounts table in accounts_ce.db. I rebooted, and tried Amazon Prime Video. Voila, everything worked.

Morals of the story

There are several morals that I’ll explicitly mention here:

Something in the fall of 2018 or spring of 2019 changed in Amazon Prime Video Android app changed (probably the name of the sync provider) that threw things out of whack in the Android sync provider configuration that neither the UI nor the APIs were ready for.
“Factory Reset” is a really poor answer to an issue like “I can’t open my app anymore”.
Root access is important. I understand the customer service anti-pattern here, but honestly, they’re our devices. Provide a “this will void your warranty” or whatever, but fixing this issue required either factory reset or root access to fix. It’s much better to let technically savvy users fix their problems. LineageOS gets this right.
The Unix philosopy rules. Text files, or even sql lite databases to control the system makes it easy to safe-heal. Don’t hide everything behind APIs on a local system.
Clear error messages allow self-maintenance. Even to me, an experienced dev, albeit without a lot of Android app experience, did not recognize this error for a while. In retrospect I feel I should have figured this out earlier, but consider the use case. I have multiple “downloaded video watching” options between Netflix, Plex, and Google Play, so this was bothersome but not a high priority for me. I fixed this on my birthday as I had blocked a good portion of my birthday to focus on low priority nagging technical issues…

Year of Linux on the Desktop

Sat, 03 Aug 2019 09:12:13 -0700

I have used Linux under one form or another since Slackware around 1994. Not sure when exactly, but the kernel was definitely pre-1.0 and ELF was not yet a thing. Since as long as I can remember, people have said “xxxx is the year of Linux on the deskop”. I haven’t found any good citations for this, but there is a reddit thread on the subject.

Linux is all around us, most predominately in servers, but also in IoT devices, our infomatics systems in our cars, our watches, our streaming devices (Roku, Chromecast, Fire sticks and the like), Amazon Echos, light switches, printers, etc. Every Android device currently runs a Linux kernel, though much of the Apple ecosystem does not.

I understand the point about Chromebooks or Android being considered a Linux desktop, but for me, a desktop includes the full freedom to change all aspects of a system, including root access, packages, etc., all things that both Chromebooks and Android try to make difficult. Historically, the things that have made Linux difficult has been hardware compability. I believe that problem has been somewhat exacerbated over the last few years as hardware manufacturers want their devices available on Chromebooks and on servers. While I’m not necessarily a fan of the GPL v2 license or the tight tie between hardware drivers and the Linux kernel, but from a market perspective I believe these two choices, plus the market dominance of Linux, has put a lot of open source pressure on hardware vendors.

In homes, a big barrier to adoption of Linux has been PC gaming. This continues to be a big barrier today, and is one of the reasons why I hesitate writing this post. I still use Windows when I want to crank up a game. In offices, the biggest barrier has been Microsoft Office. There is LibreOffice, but it just isn’t 100% compatible with Microsoft’s version. Those who know me know that I love to hate on Microsoft Office (I’ve never run into another product that more consistently does the wrong thing while trying to be “helpful”), but the quirks and sometimes crazy ancient bugs kept for backward compability sake are hard to replicate. Another hesitation to writing this post - I basically remote into a Windows box for access to Microsoft Office where I can write Office documents, though I do generally read them in LibreOffice.

So, with those two caveats, in 2018 it has been the year of Linux on the desktop for me. I have two machines for regular use. At home, I use a 2015 Macbook Pro, technically able to boot macOS, but in reality hasn’t been booted to that OS since mid-2018, and technically that wasn’t required either. I’m considering wiping the MacOS portion of the machine entirely. On the road, I use an Asus Chromebook Flip C302CA. That Chromebook can technically boot ChromeOS (with a Linux kernel), but again, this hasn’t been booted to that operating system pretty much since I got it.

It has been great, but I agree with others that this remains a bit of a “only geeks need apply” endeavor. Mac hardware compatibility required some tweaks and system setup. The Chromebook was much cleaner due to the hard work of the folks at GalliumOS, but still, there are config files and Google searches and etc., etc.

I think the best sign, and least “only geeks need apply” situation, is the presence of pre-installed Linux laptops. In definitely a particular order, the Librem 13/Librem 15 has pre-installed Linux on their devices. Here, you’re buying a premium Linux experience and making a political statement that you value security and privacy. Laptops from System76, aren’t quite as fanatical, but still have security and privacy in mind while building. However, a good sign that Linux is making slow, slow progress in the march to the mainstream is the Dell XPS 13, preloaded with either Windows or Ubuntu.

For me, I’m super happy with my current travel laptop. While it’s a Chromebook, it has no fan and charges from both sides - super useful when on a plane. I’ve had some logistics problems with meetings, but never in a typical “hey, put this on the projection screen”, for which I just pull out a Usb-C->HDMI converter like many newer laptops need, then reconfigure my display. Usually the issues come up when they want me to also share with remote participants and are using some Windows only software. This is when I pull my “remote into a Windows machine” cheater card.

At home, I’m relatively comfortable. I still have some issues with video and screen sharing in remote conferences. This is software dependent, but you might be surprised by the number of conferencing systems that support Linux directly or support web technologies. Otherwise, most stuff I do is on the web, or using Docker, which is much better on Linux.

If you’re interested at all in trying it out, Ubuntu or Mint are the most user-friendly ways to try it out. Most people I know live on the web, in which case, it really doesn’t matter which one you pick. Ubuntu is probably a bit more friendly, so if you want the “easy button”, go for that. If I were installing for someone else, I’d probably go with Mint. For myself, I think anything in the Debian family has the most supportability, and if you’re interested in a political statement, the core Debian only accepts “free” software. I’ve played with other distros, but I think they’re a bit too bleeding edge for folks looking for a “year of Linux on the desktop”. I assume most of these people are looking for “give me pre-installed, or super-easy self install, but cheaper than Windows” experience.

So, caveats about games and Microsoft Office aside, I think that Linux is much closer to the year of Linux on the desktop. The chasm has not yet been crossed, and may never be, but it’s closer than any time in the past.

Minimal Golang System

Sat, 16 Dec 2017 09:45:30 -0800

I’ve been doing some experimentation with creating bare bones systems. These come with a minimal of operational issues - fewer moving parts requires less upkeep, have less code for security issues, etc.

Golang is a fantastic language for this, as it is easy in Go to produce a statically linked binary (outside of Windows). Due to the nature of Windows some dynamic linking is necessary. A static binary (admittedly lacking C support) can be generated with a command similar to this:

CGO_ENABLED=0 GOOS=linux go build -a .

To package, we’ll focus first on the container experience. Docker is ubiquituos at this point. rkt is another alternative but this system is docker compatible so we’ll focus on docker.

The container runtime here does most of the heavy lifting for us. Containers share the kernel of the operating system, so as long as the system is statically linked, we don’t need much from the rest of the ecosystem. For a simple project we need only the kernel and possibly (probably) some networking. The runtime generally provides this, so we can get away with the following:

FROM scratch

COPY mybinary /mybinary

RUN mybinary

This results in a super-small image with very little going on, even less than alpine linux. Of course, as with all things container-based, we are dependent on the kernel used by the host OS. Sometimes, we need ssl support, which requires additional infrastructure in the container. For that, all we should need to do is copy the certs we need into the container in the expected place:

COPY ca-certificates.crt /etc/ssl/certs/

To achieve better isolation and build repeatability, we need only contain the kernel being used. This, while significantly harder, is achievable. As this will run on bare metal or a hypervisor, we will need to tailor the solution to the underlying environment.

The process was pretty well documented by earlier work, so I will instead focus on the changes I’ve made to achieve a single-use virtual machine, rather than an entire Linux from Scratch built out. Before proceeding it’s worth skimming the link above to get a feel of the process, at least up to the point of installing and configuring busybox, which we won’t need. I’ll describe the differences here.

First, sections 1 and 2 are interesting and work flawlessly but are unnecessary. If you’re working with QEMU, you can simply create the disk image, attach that image to another QEMU instance with a running Linux OS, and partition/format the block device from within that instance. There is no need for all the fancy loopback mounting and offset calculations. I feel going through the guide helped me learn a lot, but if you just want to get things done and you have a Linux VM already you can save a ton of time by simply partitioning/formatting within another VM. With this process, you can also use the grub that’s likely already in your Linux VM to copy stage1, stage2 and e2fs_stage1_5 over into the /boot/grub directory on the target device (QEMU raw image file). You can also install grub directly onto the device without going through the boot floppy procedure.

I found the most difficult part of the process to be getting the right kernel compilation options. I haven’t created a minimum configuration yet, but the firecracker VM team has done quite a bit of work in this area, so starting with their config is a good starting point.

First Thoughts on Rust

Thu, 13 Jul 2017 16:43:28 -0700

I’ve read the Rust book a year ago but never actually programmed Rust in anger until this past week. I intend to do more with Rust, but wanted to document my initial thoughts on the language as a consumer, partly for posterity, partly to avoid stockholm syndrome, and partly I think it might be useful to anyone on the Rust team interested in the out of box experience with the language.

I believe that my background and experience is relevant, as I’ll discuss later in some of these points. I’ve been coding as a student and professional over the past 30 years in a variety of languages. I’ve dabbled in systems programming, done a bit of embdedded, but for the most part stuck with application programming. With that said…

The Good

Type inference

I kept putting in type information, only later to take them out when I realized the type inference is actually pretty awesome.

Borrow checker

While I kept hearing a bunch of horror stories about fighting the borrow checker, in my experience this was not an issue. I just rolled through my coding without too much issue and did not scratch my head much over borrow checker errors.

Compiler errors

The non-issues with the borrow checker might be due in part to the clarity of the compiler errors. I found them downright enjoyable, and outside of them actually fixing my code for me, I didn’t see much room for improvement. I think the borrow checker problems people had were due in part to the old error messages, but I think the overhaul in 1.12 might have been a large part of fixing the issues as well as the perception.

Pattern matching

I love working in a language with good pattern matching. I didn’t get a chance to use it all that much (thanks partly to the ? operator), but in my small amount of use it was a delight.

The weird

unwrap()

I found I was adding unwrap() everywhere. I can’t say this is good or bad - the nice thing about it is it forced me to understand where things could blow up on me. The downside is that I had to type it, then read it, all the time. This might be an area where Rust could improve with a bit of simple syntactic sugar - just a character or something so I’m still forced to think about what I’m doing without having to read past it every 15 seconds.

Things that struck me as sub-optimal

Errors

One of the most difficult challenges was around error handling. Clearly the language itself has been struggling with this for some time. I had a small function (10 LoC or so) that I wanted to read from a file and parse the data. Either the read or the parsing could fail, and each would provide a different Error type. Rust’s handling of a situation like this frankly sucks. There is a relatively new ? operator that is actually really quite cool. It reduces a ton of boilerplate code and will basically either return an error from the function if an error occurred during the call, or it will proceed. Upon return from the call you’ll end up with a Result type that must be unwrapped with unwrap(), but otherwise Bob’s your uncle. This works great if:

You only do this once in a function
You do it multiple times with the same error type

If you don’t follow either of these, the compiler has no way to infer the Error type that could be returned. The only answer today is to write a ton of boilerplate. How much boilerplate? Well, there’s so much boilerplate that the real solution today is to skip the whole process and just use the third party crate error-chain. While this is a great library, it should absolutely not be necessary. Also consider it doesn’t solve all the boilerplate. I still needed:

#[macro_use]
extern crate error_chain;

mod error {
    use std;
    use serde_json;

    error_chain!{
        foreign_links {
            IoError(std::io::Error);
            JsonError(serde_json::error::Error);
        }
    }
}

In my code. For those keeping score, that’s 11 lines of boilerplate code, and an external crate all to manage the fact that two things might error in a function that currently consists of 7 lines of code. Folks, that be crazy.

Strings

So I have a 100 LoC program, and even there I need 3 different string types. What a pain! Here are the three types I used and why:

str. This is a Rust primitive. It’s immutable and stack-allocated UTF-8. Some functions require str.
String. This is part of std. It’s mutable and heap allocated UTF-8. Some functions require String, not str.
OSString. Also part of std. It’s immutable and “a bunch of bytes as provided by the OS”. Anything dealing with files, for instance, will need this.

At one point I literally ended up with this line of code:

rc.insert(servicename.into_string().unwrap(), newestserviceentry.to_str().unwrap().to_string());

That seems beyond silly. I get where Rust feels it’s required and I understand that fundamentally all three string types serve a unique purpose, but can we not do any better in 2010? At the very least the compiler should be able to determine whether or not a string is being mutated and “do the right thing”, thereby preserving Rust’s “zero cost abstraction” philosophy and removing one of the string types. This constantly tripped me up.

Bottom Line

I think Rust has a bright future. The icky parts are fixable and/or livable, and the roadmap is already looking to address some of the sharp edges. I look forward to working with the language, though almost certainly in a primary way, for a long time.

For those interested, the code I wrote was to process AWS CLI model files (the files used to describe all the service commands, parameters and help text - the cli itself is basically just an execution engine). It can be found as cli-model-parser on github and it’s still pretty raw but handled what I needed at the time. I’ll be doing at least a little bit more to it and might be doing a lot, depending on how things go. This write-up was based on commits up to f73e371.

Rasperry Pi Headless Bootstrap

Thu, 09 Mar 2017 14:16:06 -0800

Recently I picked up a new Rasberry Pi Zero W and was excited but also lamenting the fact that I’d have to dig out a keyboard and mouse. Being lazy, and being willing to work really hard to remain lazy, I was determined to find a way around this.

I grabbed a MicroSD card and put Raspian Jessie Lite on it. I then extended the root partition which I found easier to do before first boot since a) it was a virgin distro install and b) since I wasn’t doing it from the running system there were no reboots involved - I could simply eject the card and plug it back in to refresh the block device listing.

From there, I found a pretty good run-through of adding the ability to connect via bluetooth here: https://hacks.mozilla.org/2017/02/headless-raspberry-pi-configuration-over-bluetooth/ The article is pretty good and though the script details are a little odd I went with most of it. One big difference is that I did not allow automatic login as I want to allow bluetooth serial connections permanently (so I can add public hotspot networks later if I’m travelling). That was an easy change - just remove the “-a pi” off the end of the ExecStart=/usr/bin/rfcomm line, and it removes the need for the “Security” section near the bottom, since, sure, you can find/pair/connect, but all you’ll get is a login prompt. I followed the instructions in the article, so I won’t repeat them here, but this is the script I used (with a generic device name) without the “-a pi” on the getty line.

#!/bin/bash -e

# Display name in BT discovery list
echo PRETTY_HOSTNAME=MY-DEVICE-NAME

# Edit /lib/systemd/system/bluetooth.service to enable BT services
sudo sed -i: 's|^Exec.*toothd$| \
ExecStart=/usr/lib/bluetooth/bluetoothd -C \
ExecStartPost=/usr/bin/sdptool add SP \
ExecStartPost=/bin/hciconfig hci0 piscan \
|g' /lib/systemd/system/bluetooth.service

# create /etc/systemd/system/rfcomm.serfvice to enable BT serial from systemctl
sudo cat <<EOF | sudo tee /etc/systemd/system/rfcomm.service > /dev/null
[Unit]
Description=RFCOMM service
After=bluetooth.service
Requires=bluetooth.service

[Service]
ExecStart=/usr/bin/rfcomm watch hci0 1 getty rfcomm0 115200 vt100

[Install]
WantedBy=multi-user.target
EOF

#enable the new rfcomm service
sudo systemctl enable rfcomm

#start the rfcomm service
sudo systemctl restart rfcomm

With the script in place to allow bluetooth I needed one more thing. The terminal emulators for Android over bluetooth serial connections kind of suck, so for that reason I wanted to be able to get in and get out of the bluetooth side of things as quickly as I could, basically bootstrapping to wifi.

I therefore created an additional script new-wifi that looked like this:

#!/bin/bash
sudo iwlist wlan0 scan |grep -F SSID
read -p 'SSID:' ssid
read -s -p 'Password:' password
sudo sh -c "wpa_passphrase $ssid $password | sed '/#psk/d' \
  >> /etc/wpa_supplicant/wpa_supplicant.conf"
sudo wpa_cli reconfigure

This handles networks that are broadcasting SSIDs just fine and doesn’t expose passwords in wpa_supplicant.conf. Make sure to chmod 700 on the script. For non-broadcast SSID networks, I’ll likely need to modify the script to add the SSID as documented here: https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md

General Connection overview and Mac specifics

Next up, we want to connect to this thing. On Linux and Mac I found the easiest way was to use screen. I didn’t try without specifying the baud rate so I don’t know if that will work, but the general steps are:

Pair with the Pi
Find the device
(sometimes) rfcomm bind to the device
screen 115200

The baud rate should match the script above, so if you change it, just make sure it matches. On mac, the device just “shows up” after pairing, so you’ll look for it in /dev/cu.MY-DEVICE-NAME-SerialPort. You’ll also see a /dev/tty.MY-DEVICE-NAME-SerialPort, but you’ll want to use the cu version. There’s a great description of the differences here: http://stackoverflow.com/questions/8632586/macos-whats-the-difference-between-dev-tty-and-dev-cu

Linux

On Linux, something like this might also show up automagically. I like to use arch linux when I can since nothing is automatic and therefore I learn much more about the internals, so after pairing and making sure my system could work with bluetooth, I was able to create a serial device using rfcomm. A great overview of this process is in this gist: https://gist.github.com/0/c73e2557d875446b9603 Note that the device becomes /dev/rfcomm0, but a simple screen /dev/rfcomm0 115200 is all you need to connect. For me, I got ‘screen is terminating’ when I tried this. I then realized my user didn’t have permissions to read/write to /dev/rfcomm0. So, I used ‘sudo screen…’ until I added myself to the uucp group. The uucp group has rw permissions on the file, and you can add yourself with: usermod -a -G uucp myusername

Android

On Android I needed an appy-app to connect to Bluetooth. As I mentioned earlier they mostly suck as they’re usually optimized for non-terminal super-raw usage. The two that I found the best was BlueTerm2 (a fork of BlueTerm that uses volume buttons to send control characters instead of the jog wheel that hasn’t existed in forever), and SENA BTerm Bluetooth Terminal. I’m not linking to BlueTerm2 here because I don’t want to send folks over there. It’s mostly superior to SENA with one glaring issue - it doesn’t seem to be able to send upper case characters. The source code is available so if someone fixes it I’ll use BlueTerm2 in a heartbeat, but for now I’ll stick with SENA.

Windows

For Windows, putty should be your friend as shown here: http://www.hobbyist.co.nz/?q=bluetooth-module-device I haven’t tried this though as I don’t have a use case for it and my Windows machine doesn’t have a bluetooth adapter.

Terraform vs CloudFormation

Wed, 01 Mar 2017 14:17:24 -0800

Recently there have been discussions about the advantages and disadvantegs of using Hashicorp’s Terraform vs AWS CloudFormation for infrastructure as code on AWS. While these products change continuously, here’s a snapshot summarization of the advantages of each system.

AWS Cloudformation

Tighter integration with AWS Services: In my opinion, this is the biggest draw to using CloudFormation. You simply can’t use Terraform for things like AWS Service Catalog. Service Catalog in particular is a huge benefit to acheiving agility with control, and to avoid using it simply because your processes are Terraform-based would be a shame.
IAM-based access control: This is another example of the integration story, but if you’re already heavily invested in AWS it’s nice to not have to learn a different access control mechanism.
Has a GUI: If you’re into that sort of thing, it’s nice to not be forced to use the command line.
Feature support: For new services and features, AWS CloudFormation generally has support prior to Terraform. While there are exceptions to this, fundamentally CloudFormation has an edge as the service is inside AWS and thus has advance notice of all new services and features. Often CloudFormation support is included with the launch of a new service or feature. At the time of this writing, IPv6 VPC support is in CloudFormation but Terraform does not yet support it.

Terraform

count=n and pre-built functions: CloudFormation has some functions, but Terraform goes further, and count=n is simply fantastic.
Inspect current state: In many cases you shouldn’t need to guess about your current environment, but if you’re mixing and matching various templates or you’re the type of organization that occassionally breaks the rules and allows some manual configuration, the ability to define a template that can dynamically compensate for differences in the environment is pretty powerful.
Multiple account support: You might be able to do this in CloudFormation, but it’s complicated and a bit hackish.
Multiple region support: See above

Mythical Advantage of Terraform

One advantage I hear about Terraform but completely discard is the ability to support multiple clouds. While this is true, the reality is that each cloud works differently and makes different engineering tradeoffs, has different feature sets, etc. In the end there’s no magic bullet for deploying infrastructure seamlessly across different cloud service providers. You might be able to get away with an automated translation of one to another, but you’ll either have to architect for the lowest common denominator or you’ll lose something in translation. In the end organizations that try always end up maintaining two sets of templates.

Other approaches

I do know of some folks that are using Troposphere with some success. This provides some benefits in terms of imperative programming and the full power of Python.

Final thoughts - for now

Most people I work with get by with CloudFormation in YAML. This keeps the implementation simple and allows full integration and (mostly) faster resource support. Both Terraform and CloudFormation change often, but this is the state of things on March 1st, 2017.

Moving AWS images into Govcloud

Mon, 09 Jan 2017 15:27:50 -0800

If you have worked with AWS GovCloud, you know it is a very different region from most other AWS regions. It requires a seperate account, linked to a standard AWS account, and uses IAM users only - root users are not allowed at all. This has always been a best practice, but in GovCloud, you have no choice.

GovCloud also has fewer services than other regions. At the time of this writing, AWS Marketplace is one of the services that is missing. This creates a problem in GovCloud because many high quality AMIs are distributed for free but are only available in Marketplace.

To work around this, it is possible to move instances into GovCloud, though it is a bit convoluted. First, before doing anything, make absolutely sure you are within the terms of the license agreement moving images this way.

Once the legality of doing this has been verified, the overall process is straightforward but somewhat lengthy. Here I will cover specifics for Linux, though Windows should follow the same high level approach.

Launch Your Instance in Commercial

You will want to actually launch the instance from the marketplace. At the time of this writing, GovCloud has a single region in Oregon called us-gov-west-1. You will want to be as physically close to the target GovCloud region as possible for reasons that will become clear, so the best place to launch your instance is us-west-2 (Oregon).

Configure your instance in Commercial

Next, you will want to configure your instance. This is an optional step. As a proof of concept, I created a CentOS 7 base image from the marketplace for this purpose. As a base image, I did not want to actually configure it. What little was done (basically configuration of the ssh key) would need to be undone if I want to make it useful to others. But you can take this time to do whatever configuration you would like.

Stop your instance in Commercial

Next, we will stop this instance. What we are looking for is the underlying EBS root volume (this procedure only works with EBS-backed AMIs).

Start another instance in Commercial

Now, we can start another instance. We will need to attach two additional EBS volumes after the fact. The first must be larger than our target instance volume, the second attachment will be the target instance root volume. At this point, you will should have 3 volumes on the instance:

(root)
Large volume (xvdf)
Root volume of target instance (xvdg)

Image the target and copy to image to GovCloud

With the volumes attached (but not mounted), we will want to get an image file of the root volume. This is the whole volume, not an individual partition. Likewise, for the throwaway “large volume”, you do not really need to partition it as we will be discarding it soon. The commands look like this:

sudo mkfs -t ext4 /dev/xvdf
sudo mount /dev/xvdf /mnt
sudo dd if=/dev/xvdg of=/mnt/myimage.img bs=1M
aws s3 cp /mnt/myimage.img s3://yourgovcloudbucket/

Note that prior to the aws s3 command you should be doing an aws configure command to setup credentials (access key/secret key) for GovCloud.

At this point, you have a bit-for-bit copy of your commerical image in a GovCloud S3 bucket, so the data is in the correct region. Everything in commercial can now be terminated/deleted.

Rehydrate image in GovCloud

We will want to now launch an instance in GovCloud to finish this process. In addition we will want two additional EBS volumes in the same manner as described above:

(root)
Large volume (xvdf)
Root volume of target instance (xvdg)

The third volume should be the exact size of the original volume from the commercial region. Once launched with EBS volumes attached, we can re-hydrate the volume to make it bootable once again:

sudo mkfs -t ext4 /dev/xvdf
sudo mount /dev/xvdf /mnt
# Note you might need permissions to create/write files to /mnt
aws s3 cp s3://yourgovcloudbucket/myimage.img /mnt 
sudo dd if=/mnt/myimage.img of=/dev/xvdg bs=1M

At this point, this instance can be shut down and the large EBS volume (formerly known as xvdf) can be deleted. However, we want to create a snapshot of the rehydrated root volume of the target instance (formerly known as xvdg). From this snapshot, we can use the console or the command line to create an AMI. From the console, right click the EBS Snapshot and choose “Create Image”.

Done

And…we are done. One thing to note is that unless you have reconfigured the instance you will need to log in with the ssh key you originally specified in the commercial region as that key will now be “baked in” to the AMI.

Installation of Arch Linux on a USB stick with UEFI and legacy BIOS Support

Tue, 01 Nov 2016 19:28:18 -0700

I tend to move around machines quite a bit, especially when traveling. As such, I thought it would be useful to have a portable environment on a USB stick. Since I don’t know what type of machine I would be walking up to, this needed to support UEFI and BIOS. I wanted an actual install on a USB stick, not simply a live environment.

I chose Arch linux because I like the lightweight do-it-yourself philosophy and had heard good things about the pacman package manager. I had some previous experience with it as well and it was overall a positive experience. I remain concerned that the overall linux community doesn’t consider arch, so updates may break things. Time will tell on this.

It took a while and I learned a ton, including that certain hardware will not work with at least the UEFI on Macs. Those that are interested, btw, should use the Samsung Flash Drive Fit instead.

I created a gist of my efforts, which I’ll continue to maintain as a living document. I will not, however, include packages I install for my own use so I can keep the gist as more of a base install step-by-step.

I’m also working on making my USB multi-boot, but most other Linux distros assume during install that they should install a bootloader, etc., which would mess with my setup. I have a plan, however, so stay tuned…

Running Windows 10 on AWS EC2

Tue, 19 Apr 2016 14:11:16 -0700

Getting Windows 10 on EC2 isn’t difficult, but perusing the documentation can lead to confusion.

You can’t mount an ISO to an empty VM the way you might do in VirtualBox, so this process requires a local copy of the VM to be created, then using the aws ec2 import-image command to create the AMI. When done, not only will the image be ready for EC2, but it will be detected as Windows by AWS and be configured such that it has many of the same AWS-specific features as other Windows AMIs provided by Amazon.

Create a new Windows VM using VirtualBox
- Make sure to choose VHD as the disk image format
Install Windows 10 within the VirtualBox VM as normal. Note that a fresh install will format the VHD as MBR, which is the only partition format supported on AWS
Shut down the VM and upload the vhd file to an S3 bucket
Create a json file describing the VHD. This usually looks something like this:

[{
  "Description": "Windows 10 Base Install",
  "Format": "vhd",
  "UserBucket": {
    "S3Bucket": "mybucket",
    "S3Key": "Win10.vhd"
  }
}]

Import using the CLI: aws ec2 import-image –description “Windows 10” –disk-containers file://containers.json
The import task will have been created, but the actual import will take a while. To check progress, use:

aws ec2 describe-import-image-tasks

Once the AMI created, start instance as normal using your new ami

Update 2016-05-10: If you want to remote in, don’t forget to setup the OS to allow remote desktop prior to copying the VHD!

Consider a single container per virtual machine in production

Tue, 16 Feb 2016 13:35:02 -0800

Containers are a great unit of deployment. They’re a great way to isolate code, reduce attack areas, and, well, contain a service. When it comes to deployment in production, operational attributes of containers must be considered.

Container technology can enable significant density (described in terms of containers per vm) while retaining isolation between services. However, is this something we want to take advantage of operationally? Two significant issues with pushing for >1 container per vm come to mind.

First, for security-sensitive workloads, container technology provides a lesser security model than virtual machines on a hypervisor. A shared kernel forces us to trust user mode container orchestration tools (e.g. Docker, the daemon of which still runs as root) and kernel level constructs (e.g. cgroups and namespaces). Regardless of your feelings of the maturity level of this code, the fact of the matter is that there is simply a larger attack surface.

For this reason, only containers that implicitly trust each other can feel free to share a virtual machine in production. However, should they? The second consideration here is performance. For years we’ve been moving as an industry to more granular, single purpose virtual machines. A mail server, a web server with a single app or service, a database server, etc. Does the logic here really change with the introduction of containers? I believe the same reasons that push us toward single purpose virtual machines still hold true for containers. It becomes exponentially more difficult to deploy and manage virtual machines that contain mixed workloads. In addition, you can no longer tailor the characteristics (network, disk, ram, cpu) of the virtual machine to the workload, but need to pick general configurations.

Xen on AWS EC2

Mon, 08 Feb 2016 14:19:43 -0800

Since I’m working for AWS, I want to understand fundamentally the workings of the open source Xen Hypervisor. I also want to dig more deeply into the emerging Unikernel ecosystem. Of course, I want to do this on Amazon EC2, because generally I prefer to assume my laptop is ephemeral and could be lost, stolen, dropped, etc. However, Xen doesn’t nest well, so putting Xen in a virtual machine on top of Xen is a little bit crazy-talk.

Undaunted, I searched around, finally bumping into an old research project at Cornell called Xen Blanket that aims to do just this. The instructions are old for an old version of Xen and CentOS. So old that it took me two runs to get it right. This course lab was more explicit and ultimately I compared the directions and kind of ran through both sets of instructions together.

At the end of the day, I ended up with an older version of CentOS with an older version of Xen installed. This was all in us-east-1 (Virginia) region, and since I live in Oregon I migrated the AMI over to us-west-2 (Oregon). If you would like to play around with Xen for Unikernel development or any other reason, I’ve created AMIs in us-east-1 and us-west-2 and made them public. Once launched, use your ssh key with the ‘root’ user.

The following links will start the launch wizard in the console with these AMIs.

us-east-1 AMI (ami-fd2a0197): https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#LaunchInstanceWizard:ami=ami-fd2a0197
us-west-2 AMI (ami-bf2fcedf): https://console.aws.amazon.com/ec2/v2/home?region=us-west-2#LaunchInstanceWizard:ami=ami-bf2fcedf

index.html behavior with S3 and Cloudfront

Tue, 02 Feb 2016 16:53:08 -0800

index.html is an interesting beast in S3. S3 is an object store. It is often mistaken for a filesystem, but it is not. It is also not a web server, though it can pretend to be.

CloudFront is a CDN, and as such, it is also not a web server, though it does serve web content to users. All this makes for a strange situation for our friend, index.html.

index.html is generally used as a default document in web servers. So, if I visit https://example.com/ (and you are using SSL, right?), the web server on example.com will search the filesystem for a file named index.html in the root directory. Assuming it exists, the contents of that file will be read and sent to the browser. This behavior also applies to subdirectories, so if the directory on the server is at /var/www and you have a file /var/www/my-test/index.html, the url https://example.com/my-test/ will operate and display the contents of the file.

Because this behavior is so common, S3 website hosting mimicks this behavior. This is fantastic, but also confuses things when using CloudFront with an S3 Origin. One might expect index.html behavior to work the same way, and in a way, it does. If I setup CloudFront for the domain example.com and visit https://example.com/, the index.html file will be fetched from S3 by CloudFront and delivered as we expect. However, this behavior is a one-off by CloudFront. If I visit https://example.com/my-test/, the object in my bucket with the key my-test/index.html will not be fetched. As a result, you’ll simply get a 404.

How do we get around this? The first and easiest way is to simply create links to my-test/index.html rather than my-test/. This makes for ugly URLs though. Alternatively, we can treat S3 as a custom origin. From CloudFront’s perspective, it’s now treating S3 as it would treat any web server that might be doing any kind of processing. This works great, but comes with a couple downsides. First, you can’t lock down your S3 Bucket, so people could bypass the CDN and hit the S3 bucket URL directly. Secondly, S3 website hosting doesn’t support SSL, so you’re left with a solution that is not encrypted end to end. It’s possible, though highly unlikely, that a man in the middle could alter your content between S3 and the CloudFront edge location. I hope at some point S3 website hosting supports SSL and eliminates this problem.

Static Site Deployment with 'git push' to GitHub

Mon, 01 Feb 2016 13:40:16 -0800

The process I’ve put together for publishing this blog allows for automatic publish to the web as soon as I git commit/git push. This post describes how this is done.

As background, this blog is hosted on Amazon Web Services’ S3 service with CDN capabilities and SSL termination provided by CloudFront and Amazon Certificate Manager. This last service is extremely new, to the point that I obtained and assigned the certificate to CloudFront the very day CloudFront integration was available. As such this was a last minute change. My original intent was to use Let’s Encrypt free SSL certificates, but since ACM is also free and the service renews certs automatically, this seemed to be the better way to go.

With hosting taken care of, my next goal was to automate publishes. I keep the actual content of the blog on GitHub at https://github.com/elerch/blog. GitHub has integration with AWS SNS, so I can trigger a Lambda function based on a push to GitHub (or a commit through the web). The process looks like this: https://aws.amazon.com/blogs/compute/dynamic-github-actions-with-aws-lambda/

In my lambda function, I use the GitHub REST API to get the repo archive. From there, I can unpack the archive and copy it to S3, using Reduced Redundancy Storage to save a few portions of a cent. Outside the cost of S3, the whole process stays within free usage limits.

The lambda function, CloudFormation templates, and various scripts are hosted on GitHub: https://github.com/elerch/blog-deploy/. I’ve also created this diagram on CloudCraft:

One thing I’d really like to do in the lambda function is pull out the tight integration between the themes and the lambda function itself. However, since I don’t normally change my theme, this will wait for another day. It would also be nice to extract the actual build steps (extract archive/run hugo on extracted archive/ copy themes/copy output to S3) so that the lambda function can be used for a wider array of DevOps scenarios.

blog comments

Mon, 01 Feb 2016 11:54:14 -0800

I’ve decided to add Disqus comments to the site. Having a blog without comments is just…not a blog. That said, I’m not particularly happy with the amount of overhead it adds to the page. My base configuration (no images, no comments) involves a total of two requests to the site for full rendering (three if you count favicon business, which I don’t). The blog is delivered via AWS S3 and CloudFront, which gives me CDN capabilities. Since I’m doing static generation, the site is pretty quick. With Disqus, my request count balloons to 33 requests with a default Disqus setup. I went through Disqus and turned off as much tracking and things as I could, but I’m still at a 26 request baseline. Hopefully there are some settings to make this more minimal, since I’m new to Disqus I haven’t seen what I can tweak.

To keep the site minimal, I’ve configured Disqus only on the full post, not the summary page. As an aside, I also don’t run any tracking software (e.g. Google Analytics). First, I don’t want the overhead, and second, I don’t care to track my readers.

Moving to new site

Mon, 01 Feb 2016 11:35:00 -0800

I’ll be continuing my site off the blogger platform, but retaining this site for historical purposes. Please follow me to my new home at https://emil.lerch.org/. My first post goes into more detail about the move.

Home Backup Strategy

Mon, 07 Dec 2015 16:09:36 -0800

I’ve been meaning to document my home backup strategy for quite some time. In the process of evolving the design, I’ve tried to address the following concerns:

Rapid restoration of data in the event of an outage (RTO)
Minimal data loss from an incident (RPO)
Recovery from accidental deletes
Recovery from malicious deletes, such as ransomware
Recovery from cosmic ray damage on hard drive platters
Recovery from total destruction of the house
Recovery from a failed hard drive
Nice, but not too expensive. Contain recurring costs.

These are all interrelated, so I cannot describe a single technology or procedure that covers each item in turn. Instead, I’ll describe the solution and discuss how it addresses each point.

The solution I’ve evolved uses a local NAS with continous asynchronous backup to Code42’s CrashPlan servers in the cloud. This is the only portion of the solution that includes a recurring cost, however, that could be eliminated if I instead used their “backup to a friend” feature. The $5/month cost is reasonable, especially for unlimited data. The continuous asynchronous backup feature is nice in that it provides minimal data loss in the case of an issue. Potential data loss is limited to the time it takes any new/changed files to transfer to the Internet. CrashPlan also provides access to previous versions of files, so if ransomware encrypts my data or I accidentally delete a file, I’m covered. It can also protect me from cosmic ray damage in certain scenarios (basically, if I knew the damage had occurred). The downside, however, is that this does not protect me from cosmic rays, nor does it allow me a quick recovery time in the event of a loss of my data locally.

To deal with the as yet unaddressed concerns, I built a local NAS server using Ubuntu and the ZFS file system. I’m interested in btrfs as a replacement, but this is still experimental. ZFS has a lot of really cool features. One of the best is also in btrfs, and that feature allows stored data to reheal through checksums. This is in contrast to a RAID system which is focused on one or more drive failures. In my ZFS “pool”, I have created the equivalent of a software RAID level 5 called RAID-Z. This requires that I use 3 hard drives and can tolerate a failure of 1 hard drive at a time without losing data.

This setup has (un)fortunately been tested. When I built the array, I purchased three hard drives at the same time. This is generally not-advised, as the drives therefore come from the same batch and are therefore more likely to fail at the same time. This particular batch did have problems, but I was lucky to lose one drive at a time over the course of a year. At no time did I lose any data, although I did come close, mostly due to my own misunderstanding of the problem at the time.

Due to my use of /dev/sdb, /dev/sdc, etc naming, the drive order changed and I repartitioned the incorrect job when I already had a failed drive. I learned the hard way to use disk ids when assigning the drives to the ZFS pools. However, I learned more about ZFS and came up with another way to reduce the recovery time in the event of a loss of the array. Now my physical drives have printed labels on the outside containing the disk id so if a failure occurs it’s easy to determine the correct drive to yank from the system.

ZFS has a feature called snapshots, which are excellent for marking (and maybe reverting to) a point of time of the system. Since ZFS is a COW (Copy on Write) filesystem, snapshots do not take disk space until you start to change data. To limit the damage should I get hit with ransomware, I had already setup weekly snapshots on my ZFS. I don’t change a ton of data since the setup is mostly for pictures and videos of the kids, so I stayed with weekly. However, snapshots have another great use. You can “send” and “receive” deltas of your data between ZFS pools without having to resort to any synchronization software. Armed with this knowledge, I purchased a USB drive with enough storage space to hold my data, created a ZFS pool with the single drive (this isn’t about redundancy - this is about minimizing down time in a catastrophic loss to the main array), and I can now send my deltas to the external drive. I do not have this on a schedule yet, but I should get this cron job up shortly.

New Blog

Mon, 30 Nov 2015 09:22:24 -0800

I am transitioning to a new blog host and new, well, everything regarding my blog. I had the following goals when creating this new blog site:

Remove dependence on blogger.com
Change the url. It’s now more common to not have “blog” in the url
Reduce the page size/number of requests
Improve the speed of the site (see above, plus CDN)
SSL everywhere

This is still a work in progress and the styling will be updated moving forward, but I’m a fan of dark themes, so this is roughly where I’m going.

As part of this change, I’ve embraced hugo as a static site generator. I like hugo for a number of reasons:

It’s fast
It requires no third party software (like Jekyl requires Ruby). Important since I might author in Windows, Mac, or online
It’s written in golang, which appeals to my inner tech core
It’s easy to use, especially with github gist support in 0.15!

My old authoring process was completely online in blogger. This site is now in github, so I can still author online or offline at any time. Publishing is still a work in progress and will be fodder for another post.

Things I’m thrilled about

The size/speed of the site is fantastic. When I host the site I’ll put things on a CDN as well, but I’m glad to go from 94 requests (and console errors):

To 2:

I also get to remove the dependency on SyntaxHighlighter. This is a great library but loading it all the time when I just needed it for certain posts was a bear. Instead, I’ll be using Github Gists.

Things I’m still tweaking

Font. I wanted to avoid a web font due to the extra request, but I’m not sure that’s possible. I’m just not happy with Arial/Verdana or Georgia on the Serif side
Sizing. I increased the font size from the original gindoro theme when tweaking my fork. Since the point is to have a blog, and the point of a blog is to read it, why not make the size relatively large? Anyway, I’m still tweaking spacing and such as a result.
Flair. The site might be minimal, but maybe a bit too minimal.
Comments. I’d like to enable comments without a lot of ceremony. Currently I don’t have comments, but am considering Disqus or Discourse

Things I want improved

I’d like for hugo to support explicit image sizing and relative urls for images. On this post, I had to specify the full public url on the image tags, and height/width directives do not pass through. Since hugo is open source, that would be a great pull request in my spare time.

Probing support for architecture-specific native DLLs in .NET

Fri, 13 Jun 2014 13:50:00 -0700

Or…How I’ve tamed the Oracle beast

Over the last couple decades, very little has changed with regards to Oracle client software. Install hundreds of MB of code, update network/admin/tnsnames.ora, and finally you can make a database connection.

With a little (well, a lot) of elbow grease, I’ve made it possible for .NET applications to run as “AnyCPU” and work in 32 or 64 bit process space on either a 32 or 64 bit OS. It’s still an 84MB download (mostly due to a large dll containing all of Oracle’s error messages), but it’s bin-deployable, and only 20MB more than the combination of installing each of Oracle’s “Instant Lite” processor-specific installation packages. More importantly, since it’s bin-deployed, you can have 32 and 64 bit processes running side by side on the same machine, something that’s been problematic to do with full client installs.

The fruits of my labor are now on the the NativeProbing repository on GitHub: https://github.com/elerch/NativeProbing. The focus of the repository is not on Oracle per-se, but my goal was to achieve XCopy deployment for Oracle connectivity without massive installs or gnarly configuration.

.Net assemblies are loaded through the Fusion process, and typically follow a strict path through an AnyCPU chain, 64 bit chain or 32 bit chain. If using the <probing> element to alter flow and add platform-specific assemblies, the first time .NET hits an assembly not matching the underlying processor architecture you’ll get a BadImageFormatException. This is especially nasty; running 32 bit on a 64 bit OS is not enough to see the exception, rather, you need to be 32 bit on a 32 bit OS with your 64 bit assemblies listed first in the <probing> directory list. Therefore you need to hook into the assembly resolution process itself similar to this process: http://stackoverflow.com/a/9951658/113225

However, native libraries contain a special challenge. A different Windows process kicks in when loading a native library, and in the case of Oracle, the .Net assemblies used load native assemblies. My code on GitHub therefore uses the SetDllDirectory kernel32.dll native function to add the correct platform-specific directory to the list of paths to check. The two techniques in concert work to automatically load the appropriate DLLs.

More details of my travels are on GitHub, but the bottom line is that I have code that can augment the probing pipeline to include the correct architecture specific DLLs. This can be used for any architecture-specific DLLs with matching version numbers. SQLLite is a possible candidate for using a similar technique.

Of course, Oracle does not have matching version numbers, which is where part two of this story unfolds. To make Oracle.DataAcess.dll work, I had to resort to extreme measures. The details are on the OracleDLLHacking.md file in my repository, but the high level process was this:

Disassemble Oracle.DataAccess.dll with ildasm
Remove the public key token and change the version number in the IL
Reverse the branch logic where an exception is thrown if the assembly is “incompatible” with the native code
Reassemble the library with ilasm
Repeat for the other processor architecture

Visual Studio unit testing slow when NetBIOS over TCP/IP is enabled

Mon, 13 Jan 2014 19:01:00 -0800

According to MS Connect, this doesn’t happen. People have reported it, however, in VS 2010, 2012, and I’ve experienced the problems in VS 2013. It’s also listed at this StackOverflow question: http://connect.microsoft.com/VisualStudio/feedback/details/768230/slow-running-in-test-runner

I’ve now added this handy PowerShell command to my “initialize a new machine” setup PowerShell script:

# Disable NetBios over TCP/IP on all interfaces
# to prevent weird Visual Studio slowdowns during unit tests
Get-ChildItem hklm:system/currentcontrolset/services/netbt/parameters/interfaces | foreach{ $item = $_; Set-ItemProperty -Path ($item.ToString().Replace(“HKEY_LOCAL_MACHINE”, “hklm:”)) -Name NetbiosOptions -Type DWord -Value 2 -ea “SilentlyContinue” }

REST Basics

Fri, 12 Jul 2013 14:41:00 -0700

I’ve created a video on the basic use of REST from a consuming developer’s perspective. It covers the following:

Normal HTTP Methods:

POST
GET
PUT
DELETE
PATCH

I cover the use of PATCH to replace special-purpose functions in MVC, singular vs plural urls and search apis. I also demonstrate the use of the methods at a browser console (w/jQuery). Length - 19:23.

AngularJS and broken http options

Thu, 02 May 2013 18:19:00 -0700

Complicating the HTML5 routing and IE9 issue was the fact that AngularJS looks fundamentally broken regarding behavior around http settings. Our server offers content negotiation, so an application url for /widgets requesting a content Accept header of text/html will result in the server sending HTML for the set of widgets it knows about. If, however, the Accept type header is set to application/json, the server will respond with a json object with an array property containing widget data.

Our first problem is that a user browsing the site will want a full HTML page while Angular provides no facility (that I’ve found) for specifying an element in a full page to gather for the view. It expects an HTML fragment consisting of a single root element. To tackle this problem, we added an “X-Partial” header to let the server know to include only the partial content.

This worked fine and setting the $http settings is clearly documented…but also starts to have a code smell. The $routeProvider doesn’t seem to have a way to specify http settings, so the only way to set headers is by setting http defaults.

As it turns out, while they are named defaults, they are not, in fact, defaults. Deep in the code of AngularJS, you’ll see the following calls:

AngularJs HTML5 routing and IE9

Thu, 02 May 2013 17:05:00 -0700

We seem to be pushing the limits of AngularJS, and we’ve only started using it. The framework is very promising but definitely very new and a little rough in some areas.

We’d like to avoid hash urls in our solutions. The backend can respond to our routes just fine, and our backend framework is able to serve up the correct content (not necessarily the same content) for any area of the application. Using HTML5 mode and the history.pushState api, we can get clean URLs and no 404 errors on refresh…awesome.

Enter, IE9 (top light blue line). Our current browser standards are IE9+, Desktop/Android Chrome, FF, and IOS Safari. All browsers support history.pushState except IE9. The browser has had a recent significant drop in usage, but we can’t ignore it quite yet.

At first, we thought we could set $locationProvider.html5Mode(true) and it would either a) break in IE9 calling a function that didn’t exist (but we could define as window.location.assign(url)), or b) do a browser-based redirect based on window.location.assign. Well, RTFM: what it does is actually relatively painful. It falls back to hashbang syntax. This is similar to history.js, and IMHO, is broken in the same way as that library. Now you have URLs you can’t share between browsers without crazy (and slow) workarounds.

I tried to work around the problem by providing a history.pushState implementation. This fooled Angular into thinking it was compliant, but caused some redirect loops and ultimately didn’t work out. Next, I tried first intercepting routeChangeStart and calling event.preventDefault to no avail, then locationChangeStart also without success (I’m informed this does work, but for some reason I didn’t see it, at least not in IE9).

Finally, it hit me…since we’re not programmatically changing locations (just using anchor tags), we could simply determine if pushState is available and perform the following functions if it is not:

Avoid defining any routes with the route provider
Take our <script id=‘whateverroutewearecurrentlyon’ type=‘text/ng-template’/> that is automatically generated on the backend for the current URL and manually add it to the ng-view element.
Remove the ng-view attribute (for completeness) and add an ng-controller attribute pointing to the controller for the current route.
Disable any pre-loading for pages other than the one on which we reside.

This technique can only work in a pretty specific set of circumstances, but I think a properly designed backend system should be able to meet the criteria:

Any URL used by history.pushState actually generates appropriate content from the server.
The server at any URL provides the page layout and the application code. Providing the current page content in the script tag is nice (that’s what we do) but it’s not strictly required (#2 above could be done via Ajax).
The application code is light enough that you (and your users) can put up with full page loads in non-compliant browsers until users upgrade. I wouldn’t want to try this technique on gmail. ;-)

I reject Quora on principal, but at least now we have an answer to “Can we develop an AngularJS application which supports IE 7+ but does not have hash or hashbang urls?”.

Enabling Quartz jobs in ASP.NET applications that will run despite restart

Fri, 11 Jan 2013 16:29:00 -0800

With IIS 7.5, you can now auto-start applications and have them continuously run. However, implementing System.Web.Hosting.IProcessHostPreloadClient means having a Process method with some significant restrictions. In a Spring.Net environment, the IOC Container’s context is not yet started, and it will fail in rather spectacular ways if you try to crank it up with a hack. Even if you manage to do this, Quartz does not start up, so your jobs still will fail to run.

I spent some time on this problem and discovered that I do have access to System.Web.HttpRuntime.AppDomainAppVirtualPath, which allows us to automatically fire an initial request at the app automatically, if we’re careful. This will crank up Application_Start and the rest of the Spring (and Quartz) machinery, allowing us to keep an app running 100% of the time. This initial request, however, will be thrown away, even if fired asynchronously, if initiated during the Preload() method. I ended up using a thread to get around this problem, allowing the requests to fire 500ms after the Preload method runs.

In certain cases, we can’t just use http://localhost/yourvirtualpathhere. You might have multiple sites listening to different host headers, or you might be running SSL and don’t want to hit the application using localhost. To cover those cases, I devised a web.config appSettings scheme where additional URLs can be applied to Preload (space delimited). You can also specify an IP address or machine name, so the web.config can stay static but different URLs can be applied as you move through environments. Here is the code:

public class Preloader : System.Web.Hosting.IProcessHostPreloadClient
    {
        public void Preload(string[] parameters)
        {
            var uris = System.Configuration.ConfigurationManager.AppSettings["AdditionalStartupUris"];
            StartupApplication(AllUris(uris));
        }

        public void StartupApplication(IEnumerable<Uri> uris)
        {
            new System.Threading.Thread(o =>
            {
                System.Threading.Thread.Sleep(500);
                foreach (var uri in (IEnumerable<Uri>)o) {
                    var client = new System.Net.WebClient();
                    client.DownloadStringAsync(uris.First());
                }
            }).Start(uris);
        }

        public IEnumerable<Uri> AllUris(string userConfiguration)
        {
            if (userConfiguration == null)
                return GuessedUris();
            return AllUris(userConfiguration.Split(' ')).Union(GuessedUris());
        }

        private IEnumerable<Uri> GuessedUris()
        {
            string path = System.Web.HttpRuntime.AppDomainAppVirtualPath;
            if (path != null)
                yield return new Uri("http://localhost" + path);
        }

        private IEnumerable<uri> AllUris(params string[] configurationParts)
        {
            return configurationParts
                .Select(p => ParseConfiguration(p))
                .Where(p => p.Item1)
                .Select(p => ToUri(p.Item2))
                .Where(u => u != null);
        }

        private Uri ToUri(string value)
        {
            try {
                return new Uri(value);
            }
            catch (UriFormatException) {
                return null;
            }
        }

        private Tuple<bool string> ParseConfiguration(string part)
        {
            return new Tuple<bool, string>(IsRelevant(part), ParsePart(part));
        }

        private string ParsePart(string part)
        {
            // We expect IPv4 or MachineName followed by |
            var portions = part.Split('|');
            return portions.Last();
        }

        private bool IsRelevant(string part)
        {
            var portions = part.Split('|');
            return
                portions.Count() == 1 ||
                portions[0] == System.Environment.MachineName ||
                HostIpAddresses().Any(a => a == portions[0]);
        }

        private IEnumerable<string> HostIpAddresses()
        {
            var adaptors = System.Net.NetworkInformation.NetworkInterface.GetAllNetworkInterfaces();
            return adaptors
                    .Where(a => a.OperationalStatus == System.Net.NetworkInformation.OperationalStatus.Up)
                    .SelectMany(a => a.GetIPProperties().UnicastAddresses)
                    .Where(a => a.Address.AddressFamily == System.Net.Sockets.AddressFamily.InterNetwork)
                    .Select(a => a.Address.ToString());
        }
    }

Installing Cyanogenmod 7.2 on Verizon Droid 2 Running 2.3.4

Fri, 14 Dec 2012 10:48:00 -0800

They said it couldn’t be done. If you install 2.3.4 OTA update from Verizon, you were stuck with a version that was un-rootable. And rooting is the first step in getting a custom Android build on a phone. Specifically:

There is currently (as of May 2012) no way to root a Droid 2 with this system version.

As a word of warning, the procedures for getting Cyanodenmod are as scary as the link above, and with 2.3.4, you’re really in the deep end. Here’s a summary of what I did. I’m not providing step-by-step directions for two reasons; first, if you can’t get there from the summary, you’re probably in over your head and you have a good chance of bricking your phone. Secondly, I am providing links to my source materials with more details. I’m also taking you through the method I went, which generally used tools rather than adb commands/doing it by hand. I purchased two of the tools…I think the $8 I spent was worth it, but go read the wiki if you want to do it the hard way.

Rooting

The rooting procedures on the Cyanogenmod wiki don’t work, obviously. What you need to do is download this iso: http://www.mediafire.com/?mim304k214kl41h. MD5 is 7baee9c34f6ef7ad0b4fa219ae387c68. The forum post regarding this is here: http://androidforums.com/droid-x-all-things-root/603489-how-root-2-3-4-4-5-621-magic-md5-does-not-require-milestone-sbf.html.

The ISO can be burned to a bootable CD or USB (using Unetbutin). I went the USB route. Once booting off the USB, the scripts on the ISO walk you through what needs to be done. This was by far the easiest part of the install.

Custom Recovery ROM and Bootloader

Not having gone down these paths before, I didn’t have a conceptual understanding of how Android boot works. That turns out to be pretty important at this point, and it tripped me up for a bit.

Bootloader is run, presumably by the hardware
Bootloader looks around, checks out everything, and loads the ROM image. Depending on the environment it can run the recovery ROM or the normal boot ROM (your phone).

So, now that the phone is rooted, we need a custom bootloader because the Android one will verify that it’s the Verizon image, and we don’t want that. We need a custom recovery ROM as well…one that will let us flash new Android versions. So, the next step is to install the Droid 2 Recovery Bootstrap by ClockworkMod: http://market.android.com/details?id=com.koushikdutta.droid2.bootstrap. Also install the ROM Manager: https://play.google.com/store/apps/details?id=com.koushikdutta.rommanager.license&feature=more_from_developer#?t=W251bGwsMSwxLDEwMiwiY29tLmtvdXNoaWtkdXR0YS5yb21tYW5hZ2VyLmxpY2Vuc2UiXQ..

Use the ROM Manager to Flash ClockworkMod Recovery. Once that’s done, install the Recovery Bootstrap (remember, we need the Bootstrap to let us get into our shiny new Recovery). I missed this part and I kept going into the Android recovery…it was very confusing.

Prepping and Hacking the Cyanogen install

Download:

Cyanogen latest version: http://wiki.cyanogenmod.org/wiki/Devices_Overview#Motorola_Droid_2. Right now, this is 7.2.0, available at: http://get.cm/get/jenkins/2824/cm-7.2.0-droid2.zip
Google apps (gets you the market): http://goo-inside.me/gapps/gapps-gb-20110828-signed.zip. That is the mirror link as the primary didn’t work for me. The cyanogenmod page on this is at: http://wiki.cyanogenmod.org/wiki/Latest_Version#Google_Apps

Copy the Google apps zip file to the SD card. I don’t know if this needs to be in the root directory, but Cyanogen instructions mention putting it into root. It’s certainly easier, anyway.

**Don’t copy the Cyanogen zip file to the SD card yet. **Our last step in this path is that there is a safety check in the install to verify the kernel version so it doesn’t accidentally brick your phone. With 2.3.4 we’re safe, but Cyanogen doesn’t know this. So, you need to unzip the Cyanogen zip file above, and alter the /system/etc/check_kernel script (with a decent text editor - not notepad). Wipe out everything except the first line. Then the second line should be simply “exit 0”. More details are here: http://forum.cyanogenmod.org/topic/50469-installing-cm-from-234/#entry335479

Zip the file with the changed script and put it on your SD card from the phone.

Installing Cyanogen

Reboot the phone into ClockworkMod recovery by first powering down. Then hold power and the X key on the keyboard until it comes up. You’ll see ClockworkMod come up (maybe). I remember having an issue where nothing seems to happen. If you have that problem, hit both volume keys at the same time (if that fails, just play around with volume a bit). You’ll then see shiny new text from ClockworkMod recovery, and maybe even an error message.

At this point, it took me a while to figure out, and the ROM manager was useless for me. The directions and forums don’t know a whole lot about that error message. You’re supposed to be able to use up/down volume to arrow around the options, and power selects the option you want. However, power just made the phone go blank for me. I think what’s happening is that the droid 2 does not have /dev/tty0 and there is a bug in the recovery ROM where the power button selects /dev/tty0 as the device when pressed. However, there is another way to navigate…

What I finally figured out is that navigation can operate with the keyboard. Slide it open, and left = up, right = down (which makes sense since you have the phone in portrait mode. Up = right, Enter = enter, and Del = back. This knows where the screen is, and I’m sure it’s much more pleasant than trying to navigate around selecting files with volume keys/power.

With that working, you can perform the following steps:

Back up your existing ROM
Select and install the Cyanogen ROM
Select and install the Google Apps ROM (I think this just unpacks apks and puts them in a special directory, but I’m not sure). I thought it was way weird I was installing two ROMs.
Lastly, wipe all cache/data. I don’t remember if I performed factory reset - I think I did not. Apparently the phone will get into a “boot loop” unless this step is performed, but if you forget for some reason, you can get out of that problem: http://forum.xda-developers.com/showthread.php?t=1832130

Congrats, Cyanogen is installed! It’s like a brand new phone, so you’ll have to sign in, install apps, yada yada.

Other thoughts

I noticed the Alt Lock button does not work after I installed. Apparently this is a special Motorola hardware key and cannot be used by Cyanogen. However, pressing Alt twice will do the same thing…I’m sure that’s a default Android behavior I had no idea about. I am also missing the Motorola calendar widget terribly, but I’m sure I’ll find a suitable replacement. Cyanogen is so much better/faster. They really had me at “Add Access Point” in the wireless settings. ;-)

Update

I noticed that my WiFi does not seem to connect. It connects but the advanced settings says IP address unavailable. I tried a custom kernel but to no avail. It appears to be a dhcp problem only, and I think it is a problem with cm 7.2. I have worked around the problem by using a static IP address and DNS server, but the static DNS settings in the settings don’t seem to work. Instead I installed Set DNS and that works properly. For WiFi hotspot support I installed Barnacle WiFi tether, but I haven’t had a chance to play with it too much.

The constraint for route parameter 'httpMethod' on the route with URL '{controller}/{id}' must have a string value in order to use an HttpMethodConstraint.

Thu, 24 May 2012 07:39:00 -0700

Wow…what did that mean? Here I am, just using @Url.Action (and BeginForm and any other method that walks the route table backwards). My application is RESTful, so most routes have an HttpMethodConstraint so they only match if the method is correct.

I could not wrap my head around this message and ended up using JustDecompile to pull open System.Web.Routing and had a look at the HttpMethodConstraint Match method. You’ll note that the routeDirection parameter tells the object whether to match based on an incoming request (the normal case) or for Url Generation (used for Url.Action, BeginForm and the like).

Pulling open the source, I found this:

    switch (routeDirection1)
    {
        case RouteDirection.IncomingRequest:
        {
            ICollection<string> allowedMethods = this.AllowedMethods;
            if (func == null)
            {
                func = (string method) => string.Equals(method, httpContext.Request.HttpMethod, StringComparison.OrdinalIgnoreCase);
            }
            return allowedMethods.Any<string>(func);
        }
        case RouteDirection.UrlGeneration:
        {
            if (values.TryGetValue(parameterName, out obj))
            {
                string str = obj as string;
                if (str != null)
                {
                    return this.AllowedMethods.Any<string>((string method) => string.Equals(method, str, StringComparison.OrdinalIgnoreCase));
                }
                else
                {
                    object[] url = new object[2];
                    url[0] = parameterName;
                    url[1] = route.Url;
                    throw new InvalidOperationException(string.Format(CultureInfo.CurrentUICulture, RoutingResources.HttpMethodConstraint_ParameterValueMustBeString, url));
                }
            }
            else
            {
                return true;
            }
        }
    }

The RouteDirection.UrlGeneration case block represents what happens when the HttpMethodConstraint is asked to match for a generated Url. The InvalidOperationException represents the error message we see on the YSOD.

Even this took me a bit to work through, but the bottom line is that it’s finding the parameter name from the route and looking for the value of that parameter name from the input provided to it. Given the following route:

    routes.MapRoute("UpdateByPut",
        "{controller}/{id}",
        new { action = "Update" },
        new { httpMethod = new HttpMethodConstraint("PUT") }
    );

HttpMethodConstraint will find httpMethod (this is the value of parameterName in the code above), and look for a route value of httpMethod, expecting it to be a string. If it doesn’t, it will throw the exception. So, using the route above with the following code:

  @Url.Action("Show", "MyEntity", new { id = Model.Id })

Won’t work. Instead, you need this:

  @Url.Action("Show", "MyEntity", new { id = Model.Id, httpMethod = "GET" })

Also note that the property name httpMethod in the @Url.Action line above comes from the same property name I used in routes.MapRoute within Global.asax.cs. This is one of the few times that MVC cares about the property name used on a constraint.

Once you understand what the error message is trying to tell you it’s all pretty clear. You just need to be a rocket scientist the first time through.

Developing to Paypal

Wed, 29 Jun 2011 07:04:00 -0700

PDT
IPN
Paypal APIs:

developer.paypal.com
sandbox.paypal.com

EduSpring Part 6: Crazy Error Messages and What to do about them

Tue, 14 Jun 2011 09:36:00 -0700

Because Spring will create most of your objects up front, a simple error in the configuration Xml can have disastrous effects. This, in my opinion, is the #1 reason people fear Spring.

Here are a few error messages I’ve seen, and their corresponding solutions:

The virtual path ‘/currentcontext.dummy’ maps to another application, which is not allowed: This error message usually means you’ve deployed the application to a server, but forgotten to make the virtual directory an application in IIS. Spring performs a server.Transfer() call to “~/currentcontext.dummy” to get all the dependency wiring done correctly before another server.Transfer() brings the request back to the right place.
no application context for virtual path: This error message tells you that there’s no spring configuration setup at all.
Resource handler for the ‘web’ protocol is not defined: This message tells you that the httpModule is not active. Add the spring.net HttpModule into the appropriate section of web.config (system.web for IIS6, system.webserver for IIS7).
Could not load type from assembly: This may or may not have anything to do with Spring, so take a look a the stack trace. If you see Spring.Core (specifically Spring.Core.TypeResolution.TypeResolver.Resolve) in the trace, you can be fairly certain you misspelled a type name in the configuration.
node cannot be resolved for the specified context: Spring tried to assign an object or value to a property, but the property doesn’t exist on the object. It’s likely you misspelled the property name, or maybe you refactored, removed the property, but forgot to update the spring configuration.

Things to check if your dependency is showing up as null:

Is your configuration binding the right type? For example, if your property is a string, and you’re assigning a stringbuilder to it, Spring will just ignore the assignment.
You’re missing the object definition in the spring configuration.
You’re missing the property definition in the spring configuration.
Did you commit the cardinal sin of DI? Do not use new MyType()!

Take a look through web.config and spring.config of the Debugging Web Applications project in the eduSpring project on GitHub for troubleshooting examples. There are several errors scattered throughout the two configuration files, and the errors are fully documented.

Also note that the best way to work through the troubleshooting procedure is to correct each issue in turn, shut down the integrated web server and F5 to view the next error.

Setting up Spring.Net’s verbose logging is also shown in the web.config. You might notice that the actual project references do not include anything except standard ASP.NET DLLs. Spring, Common Logging, and Log4Net must exist in the bin directory at run time, but no project references are actually required. With VS 2010 SP1 or higher, we can put these semi-dependent DLLs in a special folder called _bin_deployableAssemblies and the build procedure will copy them in place.

EduSpring Part 5: Spring.Net in an ASP.NET environment (including MVC)

Tue, 14 Jun 2011 07:49:00 -0700

This post is part of a series on Spring.NET. I recommend starting at the beginning if you haven’t already. Also, I am walking through code in the accompanying GitHub project.

By now, you should have the basics of DI, IoC, and the benefits and drawbacks of the approach. Now, I’ll introduce you to the architecture of Spring.NET in an ASP.NET environment. I’m sure a lot of other IoC frameworks operate in a similar manner. If not, you can add code to make them work that way. ;-)

If you look at the IocWithoutSpring project, you’ll see this Main function:

static void Main(string[] args)
        {
            //These two lines are handled by the Spring.NET HttpModule
            var container = new IoCContainer();
            container.Initialize();

            // These two lines are also handled by the HttpModule by a special 
            // syntax in the spring configuration
            var service = new DoSomeWork();
            service.Worker = container.GetObject("myObject");

            // This is what the ASP.NET Framework would do
            Console.WriteLine("The output is: " + service.DoTheWork());
        }

Most of the time, creation of the IoC container itself is a single dependency that’s particularly hard to get rid of without writing some reflection-style glue code. Ideally, we want our objects to be completely ignorant of this container, though. In ASP.NET, the web.config provides us with the concept of an HttpModule, which will look at every request coming from into the web server and have an opportunity to do something with it. Taking advantage of this feature, the Spring.Net team wrote an ASP.NET HttpModule that will do just that, so the first two lines of main (instantiation and initialization of the container) are handled by the ASP.NET framework. Awesome!

Now our Spring.Net dictionary is populated, assuming we have a valid configuration. I’ll address more about the pain points of Spring configuration later, but this very early creation of lots of objects is one of the main frustrations of people who want to use Spring.Net. The next question is, what about setting up dependencies in ASPX pages?

Technically speaking, the ASP.NET framework parses an ASPX file (or MVC view) and code-generates a class. On each request, it creates an instance of this class and allows it to process the request, before destroying the object. Very stateless, but this creation and destruction of classes rubs against IoC’s Dictionary<string,object> heart.

If you read this contract from the other side (a.k.a. how would I solve this problem if I were writing Spring.Net), you can imagine yourself writing a PageHandlerFactory that can deliver the aspx page class with dependencies already injected. There are two problems you have to solve, however:

What name do you use to look up the object?
How do you deal with request-specific data?

The answer to #1 is relatively obvious if you look at the problem from the perspective of Spring. The object doesn’t really have a name, so we leave that blank, and the virtual path (~/page.aspx) is used as the type of the object since someone configuring the object will not know the type ASP.NET generates. The actual type name needs to be figured out by Spring. The answer to the second question runs a bit contrary to what I’ve been discussing so far.

In the examples so far, the initialization method of the IoC container has created all the objects in configuration and put them in its internal dictionary. However, there’s no reason that the objects need to be created right away, nor do they have to be held in the dictionary. In the case of this PageHandlerFactory, objects are created and configured at the time of the request, not at initialization. However, dependencies follow the normal rules.

EduSpring Part 4: What is so terribly broken with Dependency Injection?

Mon, 13 Jun 2011 22:23:00 -0700

Move all the CS BS to the side. This stuff doesn’t work in the real world. And it’s because of things that could have (and probably should have) been fixed by uber-geeks 10+ years ago with fancy CompSci PhDs. And I’m talking about .Net specifically here, although I can throw the same stones at Java. Most (but not all) other languages have the same problems I’ll get on my soapbox about here.

To do DI correctly, you have one of two options:

Your class has dependencies on properties/internal fields that implement Interfaces
Your class has dependencies on properties/internal fields that implement base classes

In both cases, the contracts are relatively weak. If you depend on a method that requires an object and returns an object, there are lots of things that can go wrong. And you have no way of knowing whether it will go wrong unless an until you run it - compile time checks don’t help. Here’s a short list of Murphy’s law for a single method that takes an object parameter and returns an object:

Do you assume anything about the return value? It might return what you expect, or might return null. Or, it could throw an exception. Generally, experience with DI will teach you good defensive coding, but it does take work…
Can the method handle null input values? If it errors, is it going to return null or throw an exception? If it throws, what kind of exception will it generate?
Is the method you’re calling going to muck with the object you pass in? If it does alter properties, is that a problem? What if you’re not in control of the object, and the property this dependency decides it’s OK to muck with throws an exception during the set operation?

Without some type of design by contract construct built into your language of choice, these questions become just the tip of the iceberg. Glancing through the Common Language Specification for .NET, it appears that there is no built-in construct available in the platform. C# provides a MS Research Code Contract construct in .NET 4, but they feel hacky without being part of the language, and even more hacky when implemented with IoC.

Feel free to run through the Main method for FallaciesOfInterfaces for concrete examples of how Murphy can strike. This project will compile, but will fail at almost every step. Read through the comments and fix the code one-by-one to get a sense of how, even with an IoC container and good DI practices, everything must be tested.

EduSpring Part 3: What do we need an IoC for?

Mon, 13 Jun 2011 22:05:00 -0700

This post is part of a series on Spring.NET. I recommend starting at the beginning if you haven’t already. Also, I am walking through code in the accompanying GitHub project.

Last time, I walked through why we might want to use dependency integration. Outside the authentication example, here are a few other examples:

Authentication
Authorization
SMTP settings
Payment gateways
Business rules (rule engine style)
Branding

This time, I’d like to explore how an IoC container actually does its job. By understanding how the IoC does its job, we can understand the value that it adds to our solutions. In the project “2 - IocWithoutSpring” on GitHub, you’ll see a quick, hand-coded IoC container. It’s amazingly brilliant (insert sarcasm here) and also relatively close to what a real IoC container does (really). Here is the IocContainer class in all it’s glory:

class IoCContainer
    {
        private readonly IDictionary _allObjects = new Dictionary();

        public T GetObject(string objectName)
        {
            return (T)_allObjects[objectName];
        }

        public void Initialize()
        {
            // Wouldn't it be nice if we could configure this through 
            // app.config or web.config?  Spring does that!
            _allObjects.Add("myObject", new ClassA());
        }
    }

Most IoC containers work in a similar manner. There’s some sort of initialization function, and there’s some sort of GetObject method. Internally, all it does is “new up” all the classes you’ve defined and shoves them in a global dictionary object. The rest, as they say, is window dressing. As you see in the comment, once you’ve built an IoC container, you can think of all kinds of cool things to do. It would be really nice if you could configure from web.config/app.config, or maybe a separate XML file, or maybe the database, or maybe all of them? What if you added the ability to set properties on the objects, not just create them? How about the ability to create an object based on the state of another object (Spring calls this a factory method). How about defining objects in one place, and values for properties in another? IoC containers have all this stuff, and it’s built for you…there’s no reason to reinvent the wheel.

But at the end of the day, remember: An IoC container, at its heart, is a Dictionary<string, object>. And that’s all, folks. Another key thing to remember, especially in a stateless ASP.NET scenario, is that those objects exist during the lifetime of the application. This is a performance gain (no GC stepping in, no creation of multiple objects), and a bug factory if you assume your objects are going to be created, used once, and destroyed.

More goodies are in the main file if you read the comments, but I’ll explore them more in depth in later posts.

EduSpring Part 2: Simplifying dependencies

Mon, 13 Jun 2011 17:25:00 -0700

Last time, I introduced you to Spring, IoC, and Dependency Injection. If you haven’t read that post, please do. This time, I’ll start walking through some of the code in the accompanying GitHub project. The code is organized by project in order of these posts. Some conventions (specifically 1 class/file) have been ignored specifically to let the reader go through in a linear manner. All projects are setup as console applications.

This time, I’ll tackle the first project, “0 - Advantages of DI”. This is one of the few projects I that will have more than one file to peruse. The first file is a baseline…how we would solve a particular problem in a “traditional” procedural manner. I’ve simulated a program that calls a service (MyClass) that needs to authenticate the user prior to performing some work on behalf of the caller. If authentication fails, it will throw an exception. Here is the class:

/// 
    /// Simulates a typical class
    /// 
    public class MyClass
    {
        /// 
        /// Authentication provider.  We can't change this out for the production version 
        /// without changing source code and recompiling
        /// 
        private readonly MyAuthenticationProvider authenticationProvider = new MyAuthenticationProvider();

        // Comment the line above, uncomment and recompile for production.  
        // 
        // By recompiling, you won't know if you're testing the same
        // code that exists in production (did you just change this line, or something else too?)  Versions might 
        // be different, etc.
        //private readonly MyProductionAuthenticationProvider authenticationProvider = new MyProductionAuthenticationProvider();

        public void DoWork(string user, SecureString password)
        {
            if(!authenticationProvider.Authenticate(user,password).Identity.IsAuthenticated)
                throw new SecurityException("Not authorized to perform this action");
            Thread.Sleep(3000);// Simulate some work
        }
    }

You’ll notice that when we move this class into production, we have to remember to swap out our authentication provider. In reality, the new authentication provider might now call a central authentication single-sign-on service through a web/rest service, might redirect the user in order to change password, or any number of crazy things. For these purposes, I just check that the password is “bar”.

The main point here is that we have to remember to swap, and we’ll have to recompile the application when we do that. If we forget, anyone can log in with anything and we’ll happily give up the goods. Since everything has been “tested”, it’s likely we won’t thoroughly check the service (indeed, if we’re on a production system, and this service’s purpose was to delete a bunch of accounts, we probably wouldn’t test it). After deployment have two versions of the binary - one for production, and one outside production. Who is to say there aren’t any other changes in this code. Now we have a maintenance problem too - we can’t assume the code is identical, and every time we deploy we run the risk of missing this step. But, it’s simpler when we’re coding it, isn’t it?

If we could just pass this dependency into MyClass, we’ll be in much better shape. At the minimum we can centralize the changes, and in the best case we can move these changes out into a configuration file or some other mechanism that doesn’t require code changes for this simple switch. Enter the next file in this project - “1 - WithDependencyInjection.cs”. Here are the contents:

/// 
    /// Simulates a typical class
    /// 
    public class MyClassWithDependencyInjection
    {
        /// 
        /// Authentication provider.  Because we use an interface, we no longer 
        /// need to change this code after it's been tested
        /// 
        private readonly IAuthenticate _authenticationProvider;

        /// 
        /// Constructor to establish this class' dependencies.  Since the 
        /// class is not "complete" (can't operate) without an authentication
        /// provider, we require an object up front.
        /// 
        /// public MyClassWithDependencyInjection(IAuthenticate authProvider)
        {
            if (authProvider == null)
                throw new ArgumentNullException("authProvider");
            _authenticationProvider = authProvider;
        }

        public void DoWork(string user, SecureString password)
        {
            if (!_authenticationProvider.Authenticate(user, password).Identity.IsAuthenticated)
                throw new SecurityException("Not authorized to perform this action");
            Thread.Sleep(3000);// Simulate some work
        }
    }

    /// 
    /// New Interface introduced.  Our two classes don't have a real-world inheritance relationship,
    /// but the authentication process is identical, so we build a contract
    /// 
    public interface IAuthenticate
    {
        IPrincipal Authenticate(string user, SecureString password);
    }

    class TestingAuthenticationProvider : IAuthenticate
    {
        public IPrincipal Authenticate(string user, SecureString password)
        {
            // Implementation for development
            return new GenericPrincipal(new GenericIdentity(user), new string[] { });
        }
    }

    class ProductionAuthenticationProvider : IAuthenticate
    {
        public IPrincipal Authenticate(string user, SecureString password)
        {
            if (user == "foo" && password.StringEquals("bar"))
                return new GenericPrincipal(new GenericIdentity(user), new string[] { });
            return new GenericPrincipal(new GenericIdentity(""), new string[] { }); ;
        }
    }

Here, we’ve done the following things:

Introduced an interface for authentication and marked both providers as implementing the interface. We could just as well have introduced a base class (concrete or abstract), but we have no default implementation, so an interface is probably best.
Introduced a parameterized constructor for the main class. Now this class lets the world know that it can’t do it’s job without a way to authenticate. And it doesn’t care how the authentication gets done, just that it can authenticate.

Back in the main program, we can now (sort of) demonstrate what we’ve done. Obviously without changing the code we can’t demonstrate changing the provider for MyClass, but we can demonstrate the DI approach to the problem:

static void Main(string[] args)
        {
            // Without DI
            new MyClass().DoWork("foo", new SecureString().Append("bar"));
            
            // With DI
            var testSystemProvider = new TestingAuthenticationProvider();
            var productionSystemProvider = new ProductionAuthenticationProvider();
            // The choice of test/production is now made here, outside of the class.  
            // The class shouldn't "care" who does the authentication work, it's job is to do other work
            // The new statement in the next line would be done through an IoC container (e.g. Spring) in a 
            // real environment.
            new MyClassWithDependencyInjection(testSystemProvider).DoWork("foo", new SecureString().Append("bar"));

            // Switch the constructor (in Spring.NET, done through a config file change) and now we're running against 
            // "production".  No recompile necessary
            new MyClassWithDependencyInjection(productionSystemProvider).DoWork("foo", new SecureString().Append("bar"));

            // We can also do this without a container.  Here, we'll get type information from the command line.
            // Change the provider type via project properties to see how this works.
            // The next line is basically what an IoC container would do
            var myAuthProvider = (IAuthenticate)Activator.CreateInstance(Type.GetType(args[0]));
            new MyClassWithDependencyInjection(myAuthProvider).DoWork("foo", new SecureString().Append("bar"));
        }

With dependency injection techniques, the caller can decide the authentication mechanism. Given a proper IoC container, this configuration will be centralized, but for now, we can do it manually by creating two objects, one of type TestingAuthenticationProvider and one of type ProductionAuthenticationProvider. These are passed into the constructor for the DI class, and voilà, we can change the way authentication works. In the last two lines, I show how we can even pass in the type name via the command line. Now, for the first time, we have the ability to change the way we authenticate without changing code.

…and that, folks, is all an IoC container is about.

EduSpring Part 1: Introduction

Mon, 13 Jun 2011 16:46:00 -0700

I get quite a few questions about Spring.Net, so I thought I’d put together a VS Solution, presentation, and set of blog posts to provide some background and details about what it is, why/when to consider using it, and how to configure and debug the framework.

A lot of this material will not be specific to Spring.Net, but rather Dependency Injection generally. To understand Spring.Net, you must be familiar with Inversion of Control and Dependency Injection. You can read about them on Wikipedia, but here’s a short summary:

Inversion of Control: Flow of control of a system is inverted in comparison to procedural programming
Dependency Injection: A specific technique to achieve inversion of control with respect to dependencies

What this means in layman’s terms is that instead of a program:

Starts up
Creating new services for whatever needs to be done
Using the services/doing the work
Disposing of the services and returning the results

The program now does only what it’s responsible. Namely:

Starts up
Uses services that have been configured for its use/does the work
Returns the results

This makes the program itself easier to test, simpler to code/maintain, and separates infrastructural responsibilities (determine location of services and creating them) from the business logic the program was created to deliver.

The devil in these particular details are in #2. How does one create and configure services for a program? We can code this by hand, but it’s a lot of boring glue code, and doing it right is incredibly difficult. Enter the IoC container - a piece of pre-built code that will do this work for us. Depending on the application, we don’t even need to use the container explicitly. In a system like ASP.NET, we can use HttpModules to intercept requests and wire dependencies automatically.

One significant problem with this approach are the complexity of configuration. Using Spring.net, I can have a dependency-injected factory method to deliver an instance of an object used to populate a property that will be injected into another object that will have additional advice applied through Spring.Net’s AOP module. When that fails, I might get a “node cannot be resolved for the specified context” without any additional information pointing me to the cause of the problem. With flexibility comes complexity.

Another issue is a fundamental problem with .NET (C# and VB and I believe all .NET classes adhering to the CLI). Interfaces are incomplete contracts.

Finally, with regards to Spring.Net in particular, the framework can be slow to move (IIS7 Integrated mode, introduced with Windows Server 2008 on Feb 27, 2008, gained support in Spring.Net 1.3.1 released December 10, 2010). Also, no Mono support exists as of this writing.

Next up - some walkthrough code.

Total application design: circa 2011

Sun, 05 Jun 2011 23:01:00 -0700

Working with a few recent projects I’m getting pretty close to a nice boilerplate for a “standard” web application built with the latest technologies. Subject to client restraints, here’s the stack:

UI and middle tier: ASP.NET MVC3 (using VS2010 SP1). Note that this requires .NET 4, which has some nice features (I love Tuples) but I don’t think is completely necessary otherwise. SP1 is important, which I’ll mention later. For smaller-ish applications I’ve been moving away from multiple projects in a solution, mainly because I haven’t been seeing a ton of real business logic besides authorization, and I’ve been able to productize many ancillary functions into separate DLLs. Razor view engine is used, and templates are updated to mirror Html5Boilerplate.com’s templates. jQuery is applied unobtrusively.
Data access: Fluent NHibernate. This is integrated in such a way that depending on your needs, you don’t need to add a direct reference to it, though. References are needed for access to NHibernate advanced queries and the Linq provider.
URL Rewriting: UrlRewriter.Net. This is used for css/js versioning. A URLHelper extension method appends the current version number (derived at build time from source control) for the application to the script or CSS filename, and the rewriter strips it back off. We probably could use MVC3 routes here, but it seems overkill for something that could be handled by simple Url rewriting.
Dependency Injection: Spring.net. Any DI/IOC container would work here. Again, the integration with spring is very lightweight and easily replaceable. One strike against Spring.Net is the lack of support for Mono, but this hasn’t been an issue in the environments with which I’ve been working.
Other tools: Dotless is used in a development-time capacity to generate CSS, with Chirpy (customized) to keep the development experience smooth. A handful of other custom DLLs and build tools keep the project structure relatively vanilla. Hudson is used for continuous integration.

I’m putting together a NuGet package with this build. When this is complete, I’ll post an update.

Other technologies/tools on the radar are Entity Framework (NHibernate replacement) and Castle Windsor (IoC container that supports Mono). DotNetOpenAuth is under consideration as a standard way to pull in authentication and authorization functions.

Supporting Multiple Databases in Applications

Tue, 29 Mar 2011 08:48:00 -0700

In general, having a persistence-ignorant application provides a lot of flexibility. Allowing us to easily port between various RDBMS’s, NoSQL data stores, text files, the cloud, or simply storing our data on papyrus managed and maintain by a group of beer-making monks provides us with tangible opportunities to cross-sell our (non-hosted) application to multiple clients.

As any write once, run anywhere scheme, support for multiple databases can be notoriously difficult. Consider the following SQL Server T-SQL code:

Wait…what? I just created FOO, yet SQL Server is telling me that the table foo doesn’t exist. What’s going on? In this case, the database default collation has been changed to Latin1_General_Bin, a binary collation, so FOO != ‘foo’. As a result, the following query also provides no results:

To get any data back from this table, we need to match exactly:

Great, so I’ve jacked the default installation collation in SQL Server. Not a lot of people or organizations do this? But…is this an academic exercise? If you want to support multiple databases and clients, the answer is a resounding NO. Consider:

Some organizations may want the application to work on a database with binary collation. Rare, but it could happen.
Organizations might be using another database, and not all databases are case insensitive, even by default.

If you want a portable application, #2 is your much more concerning issue. Recently I verified default installations on a number of popular database systems to determine what behavior they produced. Here are my findings (in alphabetical order):

The most curious finding is MySQL. I’d love to know what reasoning was used to come to the conclusion that Table and Column names should be case sensitive, but data should be considered insensitive. In any case, programming for the least common denominator requires handling these situations.

Ideally you’d leave the SQL to an ORM. If your application is unlikely to switch database platforms and an ORM is out of the question, I’d recommend a RDBMS-specific data access layer. If you’re application absolutely must have a generic data layer, though, you must keep this in mind as part of design.

Batch uploading RDL files to SQL Server Reporting Services 2005

Mon, 22 Mar 2010 07:10:00 -0700

There doesn’t seem to be a tool to mass-upload all RDL files in a directory, so I created one. It’s available on BitBucket here: http://bitbucket.org/emil/bulk-rdl-uploader-for-ssrs-reports/overview/

The executable is available here: http://bitbucket.org/emil/bulk-rdl-uploader-for-ssrs-reports/downloads/ReportingServicesBatchUpload.exe

This is a console application with no dependencies (it calls a web service). Call without parameters for usage information. One known bug is that the “create a folder if it doesn’t already exist” logic seems to be broken for some servers, so for best results create the reporting services folder prior to running the program.

Working with a feature branch in Subversion (TortoiseSVN on Windows)

Tue, 09 Mar 2010 08:51:00 -0800

Branching and merging is one of the most nerve-racking activities for people working with Subversion. Unlike Mercurial and other DVCS where branching and merging is commonplace, typical workflows in Subversion do not include this activity. As a result, the terminology is typically confusing, the massive number of changes can be scary, and a lot of people are so concerned about “getting it right” that they typically just avoid the practice altogether and work completely outside of source control. That, of course, is the exact opposite of what we need when working on an important design spike or critical new feature.

Conceptually, what we want with a design spike or large new feature is to:

Create a new branch for the change
Work in the new branch, committing early and often
Periodically pull in (merge) the changes from the project trunk
When complete, reintegrate the branch into the trunk

For these instructions, I assume that you have a subversion repository checked out and you’re actively working on the trunk.

Creating a new branch:

Right-click on the directory you want to branch. Pick the Branch/Tag operation.
On the branch/tag screen, enter a new branch (typically the repository has a /branches folder as well as a /trunk folder). The new branch is typically /branches/mynewbranch, where the branch name itself is a folder that does not exist in the repository.
In this scenario, you will also want to choose “switch working copy to new branch/tag”. We’ll be doing work on the new feature or design change immediately, so this tells subversion that any new commits will be on that other branch.

At this point you’ll want to take note of the revision number that had the last changes.

Work in the new branch, committing early and often

This is standard subversion behavior.

Periodically pull in (merge) the changes from the project trunk

You’ll want to keep track of the last revision you’ve pulled changes from. In the beginning, this revision number is the revision that represented the branch itself.

Pick the Merge tool off the TortoiseSVN menu.
Choose the “Merge a range of revisions” from the first screen of the wizard.
In the “URL to merge from”, choose the trunk.
In the Revision range to merge, the best approach is to be precise. Put in the last revision number that was merged (e.g. if you branched when the repository was at revision 10, put in 11-HEAD). You can leave this blank, but we want to avoid merging a change twice. You may need to look at the log to help find the right revisions.

Hit next and do a test merge just to be sure before performing the actual merge. All other settings can be left at their defaults. Note that the merge is done in your working directory, so you’ll have a bunch of local changes to test and commit on your branch.

Reintegrate the branch into the trunk

Commit all branch changes
Switch the working directory back to the trunk. This is done via the TortoiseSVN switch command.
Pick the Merge command off the TortoiseSVN menu as before. This time, however, we’ll pick the “Reintegrate a branch” option.
In the “From URL” field, put the URL to the branch you’ve finished working on.
Test the merge and if happy, perform the merge, test the changes, and commit to the trunk.

Spring.Net-enabled WCF Services available from Microsoft Ajax

Thu, 25 Feb 2010 17:39:00 -0800

Implementing Spring.NET WCF services is fairly straightforward.

Implementing MS Ajax WCF services is also straightforward, if you pick the right New Item to add from Visual Studio.

The complication comes in when you want a Spring.NET WCF service that handles calls from Microsoft Ajax controls. This method will let you add them.

Step 1. Add new “Ajax-Enabled WCF Service”

Step 2. Create your methods, test and make sure all base functionality is working. This sample is for a CascadingDropDown control from the Ajax Control Toolkit, but any control will work.

Step 3. Introduce an interface to use for the methods. Spring.Net requires an interface, and this is the crux of the problem. Here I’ve created IMyAjaxService.

Step 4. Move the [ServiceContract(Namespace = “”)] from the class to the interface

Step 5. Move the [OperationContract] tags from the methods on the class to the interface method definitions

Step 6. Change web.config endpoint contract (xpath = /configuration/system.serviceModel/services/service/endpoint) to reference the interface

Step 7: Test. The test should work, without dependency injection from Spring.Net. Now we have a WCF service that responds to Ajax, but has the interface definitions just the way Spring likes them.

Wiring in Spring

For Spring.NET to handle WCF, you need .NET 3.0 or higher, and you need Spring.Net 1.3.0 or higher. You’ll need the following DLLs available for binding (either your bin directory or the GAC):

Spring.Core.dll from the 2.0 folder
Spring.Web.dll from the 2.0 folder
Spring.Services.dll from the 3.0 folder

In the Service you’ve created, edit the markup and add Factory=“Spring.ServiceModel.Activation.ServiceHostFactory” onto the end, like so:

This will get Spring.Net into the activation pipeline for the service.

Lastly, you’ll add the new object to spring configuration. It is CRITICAL that the OBJECT ID MATCH THE SERVICE NAME FROM WEB.CONFIG. The Spring.Net documentation mentions this, but I don’t think they mention it very loudly. The type information is exactly as you’d expect: the type for the WCF service class.

Also CRITICAL: Add singleton=“false” to the object definition for Spring for your new object

Intermittent Operation Aborted Errors in IE when using MS Ajax

Wed, 08 Jul 2009 22:50:00 -0700

This issue has been plaguing our project: operation aborted errors, intermittent in nature, occurring sometimes as little as once/week. Even in IE 8 there were issues, although thankfully not the crazy dialog box we see in IE 6 and 7.

It turns out that the problem is due to a bug in Ajax itself. I won’t go into all the details, since they’re covered very well in these two blog posts:

First post with background: http://seejoelprogram.wordpress.com/2008/06/08/when-sysapplicationinitialize-causes-operation-aborted-in-ie/
Second post with bug fixes to the original bug fix: http://seejoelprogram.wordpress.com/2008/10/03/fixing-sysapplicationinitialize-again/

I didn’t particularly care for the recommended way of packaging the fix, so instead I’ve used built-in ASP.NET Ajax functionality to override the default script delivery. I also updated the fixed functions to work for all versions of Ajax (at the time of this writing - currently Ajax bundled with .NET 3.5 SP1 and below). First, here are the updated functions:

Sys$_Application$initialize:

 function Sys$_Application$initialize() {
     if(!this._initialized && !this._initializing) {
         this._initializing = true;
         var u = window.navigator.userAgent.toLowerCase(),
             v = parseFloat(u.match(/.+(?:rv|it|ml|ra|ie)[\/: ]([\d.]+)/)[1]);

         var initializeDelegate = Function.createDelegate(this, this._doInitialize);

         if (/WebKit/i.test(u) && v < 525.13)
         {
             this._load_timer = window.setInterval(function()
             {
                 if (/loaded|complete/.test(document.readyState))
                 {
                     initializeDelegate();
                 }
             }, 10);
         }
         else if (/msie/.test(u) && !window.opera)
         {
             document.attachEvent('onreadystatechange',
                 function (e) {
                     if (e && arguments.callee && document.readyState == 'complete') {
                         document.detachEvent('on'+e.type, arguments.callee);
                         initializeDelegate();
                     }
                 }
             );
             if (window == top) {
                 (function () {
                     try {
                         document.documentElement.doScroll('left');
                     } catch (e) {
                         setTimeout(arguments.callee, 10);
                         return;
                     }
                     initializeDelegate();
                 })();
             }

         }
         else if (document.addEventListener
             &&  ((/opera\//.test(u) && v > 9) ||
                 (/gecko\//.test(u) && v >= 1.8) ||
                 (/khtml\//.test(u) && v >= 4.0) ||
                 (/webkit\//.test(u) && v >= 525.13))) {
             document.addEventListener("DOMContentLoaded", initializeDelegate, false);
         }
         else
         {
             $addHandler(window, "load", initializeDelegate);
         }
     }
 }

Sys$_Application$_doInitialize():

 function Sys$_Application$_doInitialize() {
   if (this._initialized) {
    return;
   }
   Sys._Application.callBaseMethod(this, 'initialize');
   if (this._load_timer !== null)
   {
    clearInterval(this._load_timer);
    this._load_timer = null;
   }
   var handler = this.get_events().getHandler("init");
   if (handler) {
    this.beginCreateComponents();
    handler(this, Sys.EventArgs.Empty);
    this.endCreateComponents();
   }
   if (Sys.WebForms) {
    if (this._onPageRequestManagerBeginRequest) this._beginRequestHandler = Function.createDelegate(this, this._onPageRequestManagerBeginRequest);
    if (this._beginRequestHandler) Sys.WebForms.PageRequestManager.getInstance().add_beginRequest(this._beginRequestHandler);
    if (this._onPageRequestManagerEndRequest) this._endRequestHandler = Function.createDelegate(this, this._onPageRequestManagerEndRequest);
    if (this._endRequestHandler) Sys.WebForms.PageRequestManager.getInstance().add_endRequest(this._endRequestHandler);
   }

   if (this.get_stateString){
    var loadedEntry = this.get_stateString();
    if (loadedEntry !== this._currentEntry) {
     this._navigate(loadedEntry);
    }
   }
   this.raiseLoad();
   this._initializing = false;
  }

Sys$_Application$_loadHandler():

 function Sys$_Application$_loadHandler() {
  if(this._loadHandlerDelegate) {
   Sys.UI.DomEvent.removeHandler(window, "load", this._loadHandlerDelegate);
   this._loadHandlerDelegate = null;
  }
  this._initializing = true;
  this._doInitialize();
 }

Next, here is how I overrode how the framework delivers the script:

  <ajaxtoolkit:toolkitscriptmanager runat="server" enablepartialrendering="true" id="ScriptManager">
       <scripts>
           <asp:scriptreference name="MicrosoftAjax.js" assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" path="~/js/MicrosoftAjax-withFix.js">
       </scripts>
  </ajaxToolkit:ToolkitScriptManager>

You do not need to use the ToolKitScriptManager - you can also do it this way:

   <asp:scriptmanager runat="server" enablepartialrendering="true" id="ScriptManager">
       <scripts>
<asp:scriptreference name="MicrosoftAjax.js" assembly="System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" path="~/js/MicrosoftAjax-withFix.js">
       </scripts>
   </asp:ScriptManager>

Note that you can replace ~/js with the directory of your choosing.

Be aware that when the framework loads the file, it automatically adds .debug or .release onto the end of the file name, so the final file reference provided will be either MicrosoftAjax-withFix.debug.js or MicrosoftAjax-withFix.release.js.

If you don’t want to bother incorporating these fixes into .NET 3.5 SP1 Ajax files, feel free to download them from here:

Debug version: http://lerch.org/js/MicrosoftAjax-WithFix.debug.js
Release version (simply a minified version of the debug js - run through YUI Compresssor): http://lerch.org/js/MicrosoftAjax-WithFix.release.js

Javascript Hacks: Using XHR to load binary data

Tue, 07 Jul 2009 10:56:00 -0700

I recently needed to get image data from a server using Javascript, base64 encode it, and post that data back to an application. While the details of why I needed to do this are a bit complex, I believe that getting image data through an XMLHttpRequest object and base 64 enconding it will become more valuable in terms of client-side image manipulation using the data URI scheme for image tags.

This would allow a Javascript developer, for instance, to load an existing image (say, a photo), without base64 encoding it on the server, load it into an image tag with a data URI, and make direct manipulations on that image.

Unfortunately, this area is relatively new and browsers have a lot of differences. Data URI support is still very new, inconsistent, and limited. In the meantime, here is how you get that base64 encoded image in the first place:

Internet Explorer:

IE has a property of XMLHttpRequest object for binary data ResponseBody. This contains exactly what we need, but unfortunately the property is not visible to Javascript. Since the string returned to Javascript by ResponseText will be terminated at the first null value, we must use ResponseBody. This requires a bit of VBScript, which can do one of the following things:

Get the numeric value of each unsigned byte and turn that into a number in a string of comma delimited numbers. This is less efficient, but gets you in and out of VBScript as quickly as possible, allowing a generic base64 encoding routine. This is the route I followed (it may be less efficient, but it pales in comparison with the XHR request just made):
```
Function BinaryArrayToAscCSV( aBytes )
 Dim j, sOutput
        sOutput = "BinaryArrayToAscCSV"
 For j = 1 to LenB(aBytes)
  sOutput= sOutput & AscB( MidB(aBytes,j,1) )
  sOutput= sOutput & ","
 Next
 BinaryArrayToAscCSV = sOutput
End Function
```
Base 64 encode it directly in VBScript.

Once this is done, we can then base64 encode it using a fairly generic function in Javascript:

Base64 = {
 
 // private property
 _keyStr : "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=",

 encodeBinaryArrayAsString : function(input){
  var ascArr;
  var output = "";
  var bytebuffer;
  var encodedCharIndexes = new Array(4);
  
  var inx = 0;
  ascArr = input.substring("BinaryArrayToAscCSV".length, input.length - 1).split(',');
  while(inx < ascArr.length){
   // Fill byte buffer array
   bytebuffer = new Array(3);
   for(jnx = 0; jnx < bytebuffer.length; jnx++)
    if(inx < ascArr.length)
     bytebuffer[jnx] = parseInt(ascArr[inx++]); 
    else
     bytebuffer[jnx] = 0;
     
   // Get each encoded character, 6 bits at a time
   // index 1: first 6 bits
   encodedCharIndexes[0] = bytebuffer[0] >> 2;  
   // index 2: second 6 bits (2 least significant bits from input byte 1 + 4 most significant bits from byte 2)
   encodedCharIndexes[1] = ((bytebuffer[0] & 0x3) << 4) | (bytebuffer[1] >> 4);  
   // index 3: third 6 bits (4 least significant bits from input byte 2 + 2 most significant bits from byte 3)
   encodedCharIndexes[2] = ((bytebuffer[1] & 0x0f) << 2) | (bytebuffer[2] >> 6);  
   // index 3: forth 6 bits (6 least significant bits from input byte 3)
   encodedCharIndexes[3] = bytebuffer[2] & 0x3f;  
   
   // Determine whether padding happened, and adjust accordingly
   paddingBytes = inx - (ascArr.length - 1);
   switch(paddingBytes){
    case 2:
     // Set last 2 characters to padding char
     encodedCharIndexes[3] = 64; 
     encodedCharIndexes[2] = 64; 
     break;
    case 1:
     // Set last character to padding char
     encodedCharIndexes[3] = 64; 
     break;
    default:
     break; // No padding - proceed
   }
   // Now we will grab each appropriate character out of our keystring
   // based on our index array and append it to the output string
   for(jnx = 0; jnx < encodedCharIndexes.length; jnx++)
    output += this._keyStr.charAt(encodedCharIndexes[jnx]);     
  }
  return output;
 }
};

Firefox:

Firefox works a little differently, as there is no RequestBody property. In this case, RequestText is not truncated as long as you override the mime type coming from the server, forcing Firefox to pass the data unaltered. All we need to do is compensate for binary data coming back and being placed in a Unicode Javascript string. To compensate, we can AND each character with 0xFF to throw away the high-order byte (see https://developer.mozilla.org/En/Using_XMLHttpRequest#Handling_binary_data). The resulting encoding function looks like this:

Base64 = {
 
 // private property
 _keyStr : "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=",

 encodeBinary : function(input){
  var output = "";
  var bytebuffer;
  var encodedCharIndexes = new Array(4);
  var inx = 0;
  var paddingBytes = 0;
   
  while(inx < input.length){
   // Fill byte buffer array
   bytebuffer = new Array(3);
   for(jnx = 0; jnx < bytebuffer.length; jnx++)
    if(inx < input.length)
     bytebuffer[jnx] = input.charCodeAt(inx++) & 0xff; // throw away high-order byte, as documented at: https://developer.mozilla.org/En/Using_XMLHttpRequest#Handling_binary_data
    else
     bytebuffer[jnx] = 0;
   
   // Get each encoded character, 6 bits at a time
   // index 1: first 6 bits
   encodedCharIndexes[0] = bytebuffer[0] >> 2;  
   // index 2: second 6 bits (2 least significant bits from input byte 1 + 4 most significant bits from byte 2)
   encodedCharIndexes[1] = ((bytebuffer[0] & 0x3) << 4) | (bytebuffer[1] >> 4);  
   // index 3: third 6 bits (4 least significant bits from input byte 2 + 2 most significant bits from byte 3)
   encodedCharIndexes[2] = ((bytebuffer[1] & 0x0f) << 2) | (bytebuffer[2] >> 6);  
   // index 3: forth 6 bits (6 least significant bits from input byte 3)
   encodedCharIndexes[3] = bytebuffer[2] & 0x3f;  
   
   // Determine whether padding happened, and adjust accordingly
   paddingBytes = inx - (input.length - 1);
   switch(paddingBytes){
    case 2:
     // Set last 2 characters to padding char
     encodedCharIndexes[3] = 64; 
     encodedCharIndexes[2] = 64; 
     break;
    case 1:
     // Set last character to padding char
     encodedCharIndexes[3] = 64; 
     break;
    default:
     break; // No padding - proceed
   }
   // Now we will grab each appropriate character out of our keystring
   // based on our index array and append it to the output string
   for(jnx = 0; jnx < encodedCharIndexes.length; jnx++)
    output += this._keyStr.charAt(encodedCharIndexes[jnx]);
  }
  return output;
 };

Ideally we’d combine these two functions into a single encoding function, but I’ve left them separate for clarity. Note also that these techniques do not appear to work for Safari, Chrome or Opera. It should work for IE6 if using the correct ActiveX XHR object, but I was not supporting IE6. I did a spot check on Safari/Chrome/Opera and they were not working, but I did not investigate as they were not supported browsers for my implementation. The actual XHR function I used was:

LoadBinaryResource = function(url) { 
  var req = new XMLHttpRequest();  
  req.open('GET', url, false);  

  if (req.overrideMimeType)
    req.overrideMimeType('text/plain; charset=x-user-defined');  
  req.send(null);  
  if (req.status != 200) return '';  
  if (typeof(req.responseBody) !== 'undefined') return BinaryArrayToAscCSV(req.responseBody);
  return req.responseText;  
} 

LoadBinaryResourceAsBase64 = function(url) { 
  var data = LoadBinaryResource(url);
  
  if (data.indexOf("BinaryArrayToAscCSV") !== -1)
    return Base64.encodeBinaryArrayAsString(data);
  else
    return Base64.encodeBinary(data);  
}

It doesn't get much better than this

Thu, 25 Sep 2008 09:13:00 -0700

How’s this for a setup message? Kinda hard to comply with…

One more technical analysis link

Tue, 27 May 2008 08:56:00 -0700

I bumped across Stoxline during my recent foray into the technical analysis world. It provides a good quick summary of some key price points.

Excercising INTC stock options

Fri, 23 May 2008 07:50:00 -0700

So I’ve excercised the remaining stock options from Intel that remain above water. There are still a large number of options that will die worthless (my original options were priced at $67 back in 2001, current stock price is $23.52, and there were no splits in between). The next set of options that even have a shot at becoming in the money are priced around $26, and with 3 weeks out, it would be surprising if they became valuable. Here’s the quick story of how the excercise went down.

I excercised in three batches. The first batch I got totally lucky and sold at $27.98, exactly $0.01 off the 52 week high. Dumb luck. I sold only half because word on the street was that Q1 would be great, and the price would keep going up. In retrospect, I think selling half was a good move, but my big mistake here was not to have sold the most expensively priced options. Those options at $26 would have actually been worth something.

The stock tanked this year, but recently began to gain ground. Oddly, it picked up momentum based on almost no news (the overall market was also up). Intel did get an upgrade that did pretty well for the stock. Due to what seemed to be a short term movement and the imminent expiration of my options, I dug out some of my technical analysis knowledge and went to town:

I watched the following indicators:

MACD: I was looking pretty short-term (<1>
Bollinger Bands: I thought this would be a fairly useless indicator, but in fact, it was just when the stock crossed the upper band that it turned negative.
RSI: Fairly useful indicator in this scenario
Slow Stochastic Oscillator: Fast stoch always seems too touchy for me. Maybe if I were trading several times a day it would be different, but the slow stoch was useful
Parabolic SAR: My primary source of when the options should be sold.

When the stock hit 25 and retreated a bit to 24.60, I decided to sell my second batch, a fairly small set of options that were priced in the mid 23’s. My primary thinking here was to lock in some of the options since going up over 5% in one day on no news was just crazy.

At 25, the MACD, Bollinger Bands and RSI indicators all went bearish, so I thought I’d sell when the price started to dip. The next day it opened pretty low and then dropped below 24.13, which was the Parabolic SAR trigger point, so now all the indicators I was tracking went bearish. I sold at 24.08, and so far, it’s down another 0.85 from there.

Lessons learned? The indicators can work…don’t rely on just one, and make sure that you use the right indicator for your goal.

Another couple links of use:

Scottrade chart: This chart is continuously updated, and summarizes what the indicators are saying (e.g. Parabolic Indicator is bearish/MACD is bullish). The conintuous updates made it more useful than Yahoo. It does, however, force you to re-add the indicators - no deep link is possible.
Yahoo chart: I was originally using this chart from Yahoo. Since the URL includes all the options, it looks exactly as I was using. However, it only updates at the end of each day.
TD Ameritrade Command Center 2.0: A Java app available for TD Ameritrade customers, it let me watch the stock in real-time during the day.

It could have been so much more effective with a little B&E

Wed, 21 May 2008 07:13:00 -0700

A couple days ago I got a flyer on my door advertising a security system and monitoring package. I’m not sure how many people actually respond to a blanket flyer like that, but it got me thinking…wouldn’t the flyer have been more effective had they managed to leave it somewhere inside the house while we weren’t home? Illegal - yes. Effective: definitely.

Check for valid stored procedures

Fri, 02 May 2008 09:23:00 -0700

I just posted a small utility on CodeProject to check for valid stored procedures, views, and functions (in SQL Server). It’s actually a polish of some work someone else had done earlier, but if you have a large number of objects and are doing significant database refactoring, you may want to check it out.

Street signs

Thu, 01 May 2008 08:10:00 -0700

When crossing the street and seeing one of the standard walk signs (such as the one to the left), I have a habit of imitating the sign and saying “white man walking”.

Well, Sweden is now crossing the gender barrier with their new signs. Where this will lead, I’m not sure. Non-caucasians? Dogs? Cats? It does make me wonder, if the signs are specific, is Sweden now implying that where this sign is posted, only women are allowed to walk?

Google Maps update

Thu, 17 Apr 2008 07:06:00 -0700

I just noticed that Google Maps introduced the ability to show historical “average traffic” for any given time on particular day of the week. I just used it to confirm my commuting schedule, and it seems to correlate well with my experiences and validate that the times I drive are pretty reasonable.

There are definitely some systemic traffic slowdowns in the Portland area. For example, 26 west bound between Murray and Bethany is nearly always a problem from 4PM-7PM, probably stemming from the fact that the road moves from 3 lanes down to 2 just after the Bethany exit. This is especially unfortunate since the third lane was just added…although I do admit that there are much more serious traffic concerns in the area to focus our resources on.

It would be nice to know that those that plan road improvements take a look at this type of data to prioritize their efforts.

Calendar sync bliss (finally)

Wed, 02 Apr 2008 19:56:00 -0700

Finally, a calendar sync solution that works. After my experiment with gSyncIT, I just couldn’t quite get past some of its limitations. Once multiple calendars get involved, things get a bit ugly.

A coworker recommended SyncMyCal as a way past those issues, and I’m pretty happy. You’ll see my calendaring setup to the right.

Google Calendar is pretty much my primary calendar, and I have several calendars loaded. Some calendars, like US, UK, Indian Holidays and the Winterhawks game schedule are information only. However, I have my own calendar, a family calendar for events the whole family is attending, and two TripIt calendars (one for my wife, one for me) that do a great job representing any travel I’m doing.

I’d like to make sure that these calendars are available (at least in free/busy form) to other Exchange users at work, and that the calendars get posted to my cell phone using Windows Mobile. The Outlook 2007 Internet Calendars feature doesn’t allow this, so I needed a solution that would allow my Google calendars to merge into my single Outlook calendar. I was also looking for my work calendar to post out to Google so I could have visibility there as well.

SyncMyCal had the answer through the use of category synchronization. Each Google calendar has their events assigned to a specific category, and only those events are sync’d with that calendar. Anything not assigned to a category (work stuff) gets sync’d to Google under a new “Work” calendar. I didn’t need full synchronization here, but it’s nice to know I have it if I need it.

What’s missing? I can’t find a way to have SyncMyCal make all Google-sourced events private on the Outlook side (although it does sync the private flag). I have to remember to mark anything truly private as such. Not a big deal, but it’s worth mentioning.

There is definitely whitespace for an entrepreneur to simplify calendaring…I can’t believe that my needs are *THAT* special…

C# Covariant Generics

Tue, 01 Apr 2008 08:28:00 -0700

Sorry for the highly technical posts…lighter stuff to come!

This has really been bugging me on my current project. In an object oriented system, if one class inherits from another (e.g. “Cat” inherits from “Animal”), and a method expects to receive the base class as a parameter, it is safe to send in a subtype. In my example, a method operating on “Animal” can take “Cat” safely. This is sometimes called “downcasting”.

This behavior should extend to generic types. Again, in my example, if a method expects a List of Animals, I should be able to pass in a List of Cats. However, C# does not allow this behavior (but the CLR does). Bah!

Here is some more background I’ve dug up on the issue:

Yet more SQL 2005 fun: Developing search procs

Thu, 27 Mar 2008 08:47:00 -0700

A common pattern in applications is to have a search stored procedure work as the backend for a search screen. If a term has not been passed by the user, the intent is that there is no filter on the results for that parameter. Here’s a great article discussing the ways that this can be acheived. I’m a personal fan of the COALESCE statement talked about in the static discussion.

One area that was not discussed in the article was the idea of having similar functionality with a list of filters (e.g. filter to these specific books in the library). I found a pretty good article talking about SQL 2005’s use of Xml parameters to do this - Xml’s main benefit being that it’s more designed for this use case than a delimited string. However, the same problem now applies…what if I either have a list of books, or don’t pass anything, meaning I want to search across all books?

My solution was to grab all the data from the reference table and build my own Xml string. Then I could use it in the main select statement:

IF @param is null
SET @param =
(SELECT 1 AS Tag,0 AS Parent,MyElementName AS [Root!1!string!element]
FROM sourcetable FOR XML EXPLICIT)

SELECT…
WHERE…
…AND
FieldName IN
(SELECT Field.value(’.’,‘VARCHAR(max)’) FinalName
FROM @param.nodes(’//string’) AS Table(Field))

I’m sure this wouldn’t perform well with large tables, but my reference tables are fairly small (<100 rows each), and it works great.

FYI: Here is an example COALESCE statement for the standard filters:

table LIKE ’%’ + COALESCE(@Name, fac.FacilityName ) + ’%’

More SQL 2005: SqlDependency update

Thu, 27 Mar 2008 08:24:00 -0700

While SQL Server data update notifications through the SqlDependency object seemed like a great idea, I now believe the architecture is fundamentally flawed…I’m looking forward to changes in this area to make the feature more robust in the next release(s) of SQL Server and/or the .NET Framework. Here are some of the problems I’ve run into:

Massive amounts of required permissions in the default run mode
Query types available are incredibly restrictive
Ineffective tear down of resources, again, in default run mode, with no programatic workaround. (The worst part? The problem is actually most acute while doing active development…)
Severe ramifications for any issue, whether caused by environment or poor coding
Complex (and maybe unusable) setup for custom service/queue implementation to alleviate the first two issues. I never really even figured out that setup when operating in a non-default schema in a restricted permission environment.

This feature is a great concept, but with an implementation that severely limits its use. My recommendation is to avoid this functionality in nearly all cases. In my current project, I’ve moved to a custom cache implementation that checks last modified date across all all lookup tables and clears the cache of all out of date lists. If something happens in between checks, we’ll just have to deal with that.

Database permissions for SQLDependency

Thu, 20 Mar 2008 08:35:00 -0700

While I like the idea of SQL 2005 Query notifications, the setup restrictions and instructions are fairly opaque. Blah! I did manage to get it working after noting all the restrictions on the query in this MSDN article, but then I made the mistake of removing dbo permissions from the user, and was thrown into the mix again for another hour of churning.

This blog post was pretty useful, but didn’t go quite all the way. Later, I found a post describing more details after some searching, and came up with this set of grant statements to make it work:

GRANT ALTER ON SCHEMA :: [schemaname] TO [Role]
GRANT CREATE PROCEDURE TO [Role]
GRANT CREATE QUEUE TO [Role]
GRANT CREATE SERVICE TO [Role]
GRANT REFERENCES on CONTRACT::[http://schemas.microsoft.com/SQL/Notifications/PostQueryNotification] to [Role]
GRANT VIEW DEFINITION TO [Role]
GRANT SELECT to [Role]
GRANT SUBSCRIBE QUERY NOTIFICATIONS TO [Role]
GRANT RECEIVE ON QueryNotificationErrorsQueue TO [Role]
GRANT IMPERSONATE ON USER::DBO TO [Role]

Windsor Solutions

Tue, 18 Mar 2008 08:20:00 -0700

After 7 years at Intel, I’ve have now joined Windsor Solutions, a 35 employee company focused on systems to manage environmental reporting and compliance at various state and local governments. It’s energizing to be part of such a vibrant, growing organization, with the ability to wear many hats!

Javascript/PHP annoyances

Wed, 12 Mar 2008 08:21:00 -0700

I ran across a problem the other day in a PHP script, where a variable wasn’t getting populated. It turned out that I misspelled the variable name - clearly my fault. However, it got me thinking about the language itself, and how it is actually somewhat hypocriful. On one hand, PHP (and Javascript for that matter), is relaxed when it comes to variables. There is no strong typing, and in PHP, you don’t even need to declare a variable before using it. On the other hand, both languages are type sensitive.

It seems to me that if you’re going to have a laid back policy in regards to variable naming, you should also be laid back when it comes to case. Being case sensitive with weak controls on variable definition doesn’t make sense. It’s the computer science equivelent of mixing first and third person.

Add-in manager not disabling add-ins in Visual Studio?

Fri, 07 Mar 2008 11:15:00 -0800

I stumbled across this problem today, and found an interesting feedback page to Microsoft regarding the issue: http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=105560

Workaround:

Regedit: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\8.0\AddIns
Go into the add-in key you’re interested in disabling
Manually set the LoadBehavior entry to 0x0

Google Calendar Sync

Thu, 06 Mar 2008 10:06:00 -0800

Looks like Google just released a tool to synchronize Google Calendars with Outlook. Woo hoo! Unfortunately, it will only sync with your primary calendar, so in the meantime, I’m still using gsyncit.

Tip of the day using 7-Zip

Wed, 05 Mar 2008 07:28:00 -0800

I just stumbled across some interesting behavior in 7-Zip. Not sure why this is how it works, but if you drag and drop a file from 7-Zip into a directory to extract, the extract will first go to the temporary directory and then be copied over into the destination file. If you click the “Extract” button and select the directory, the file is extracted directly in the destination directory.

LCD vs. Plasma

Tue, 04 Mar 2008 08:20:00 -0800

FIOS TV

Tue, 04 Mar 2008 08:19:00 -0800

Jabra BT8010

Tue, 04 Mar 2008 08:19:00 -0800

Logitech MX 5000/Bluetooth

Tue, 04 Mar 2008 08:19:00 -0800

Uniball 207

Tue, 04 Mar 2008 08:09:00 -0800

I had a little pen issue - a few of my good pens were wearing out. Bad pens kind of bother me. I’m too cheap to spend a lot of money on pens. I did a quick search on the Internet, came across this review, and was sold. Costco carries a pack of 12 for $15.68 online (I thought they were 11.59 in the store, though). Done deal - after about a month, I’m still very happy with the choice.

(Note: I do not participate in paid review stuff…my opinions are 100% uninfluenced).

...And, we're back!

Fri, 29 Feb 2008 15:59:00 -0800

What a crazy few months - more on that later. Just wanted to mention that I do intend to pick up the blog again (roughly daily as I did last year). Future topics:

Work-related stuff
Javascript/PHP hypocrisy
Consumer electronics (FIOS TV service, Logitech MX 5000 for BT, Jabra BT8010, etc.)
Work done to the house
Other stuff on my mind that no one probably cares about

Also wanted to quickly mention a big CONGRATULATIONS to Doug and Lisa Downey on the birth of their daughter Greta Marie!

Standards...or lack thereof

Wed, 29 Aug 2007 15:42:00 -0700

I was just typing up an email, and noticed that I looked down at the keyboard to find the delete key. I asked myself why, and then looked at the keyboards around me. Home, End, Delete, Insert, page up and page down are all grouped together, but on every keyboard I use, they’re in a different configuration.

My Dell keyboard at work is:

Home PgUp
End PgDn
Delete Insert

My Thinkpad:

Insert Home PgUp
Delete End PgDwn

My new keyboard at home:

Home End
Insert PgUp
Delete PgDwn

All three configurations make sense (especially the Thinkpad, which has an additional complexity of space issues), but it seems that at least on the desktop, there should be a standard.

Determining internal vs. external email

Sun, 26 Aug 2007 18:57:00 -0700

I’ve been trying to get a good Outlook filter to determine internal from external mail. You’d almost think this is a standard request, but it’s actually pretty difficult, especially at a large organization like Intel. After some time trying to figure it out and following some bad advice from Microsoft, I ran across Ray Jezek’s Blog : Outlook Rules with Exchange.

The idea is fairly sound, but alas, does not work at Intel, where we have many Exchange servers. When a message moves from one server to another, the message gets Internet headers, so the idea of filtering a message header for the word “Received” doesn’t work.

I tried other variations on this theme, though, and finally came up with the solution. My rule now matches any message with a message header containing the text of one of the servers listed in Intel.com’s MX records. Internally, our mail will not use one of these hosts, so that’s the only sure-fire way to catch external, and only external email that I’ve found. I suppose if the sender listed the IP address of the mail host my rule would fail, but that’s a risk I’m willing to take. ;-)

Slowdown in Visual Studio

Wed, 15 Aug 2007 16:24:00 -0700

The other day I had a sudden and mysterious slowdown in compile times within Visual Studio. When I didn’t see an associated CPU spike during the compile, I increased the verbosity of the compiler and checked the output, only to see it pausing at a file copy.

Here I had added my project’s SQL Express database to a library directory rather than the web site, so studio dutifully copied the database over. All this was fine, except that during development, I bulk loaded a ton of data, and the file was now about 240MB.

Note to self…don’t copy 240MB database files as part of your compile. ;-)

Google reader

Tue, 14 Aug 2007 16:19:00 -0700

So I’m now giving Google Reader a try, for the following reasons:

I am going on an extended, paid leave from Intel shortly, and will likely not be using my notebook much
As a result of #1, I want to be able to read my feeds anywhere
I believe Outlook RSS is contributing heavily to random slowdowns
Outlook RSS is certainly helping Outlook slam my hard disk (this is a problem in Outlook 2003, seems to have gotten worse in 2007, and is even crazier with several RSS feeds)
I don’t see duplicate posts in Google reader the way I was in Outlook. In Outlook it once got so bad that my RSS feeds were consuming more than 250MB of storage in my account, mostly due to one 2MB post that it refused to detect as a duplicate.

I tried Google Reader a while ago and wasn’t a big fan of the interface, but now that I’m using it a bit more I like it. A big problem for me is that it defaults to expanded view, which I still don’t like. List view is much better, as I rarely read the items (the trend tools tell me that over the last few days I’ve read 10% of posts in even my favorite feeds).

Another problem is offline usage, which was somewhat fixed by Google Gears. I don’t like the fact that I have to manually click the icon to download data (apparently this is supported in gears, but not implemented by the reader developers…fix this!). I suppose I’ll have to deal with this, but it’s one more thing to remember if I’m taking a flight.

On the flip side, I love the fact that I can click on a folder and see all the posts for all the feeds underneath it. I couldn’t do this in Outlook (and only sort of could in some other products I tried), and resorted to using Yahoo! Pipes to combine a bunch of feeds that weren’t updated frequently. The problem is that I couldn’t clearly see the source of the post anymore, since pipes renames the source to the name of your pipe, and I couldn’t figure out how to undo this.

The folder aggregation feature allows me the ability to undo the work in Pipes, and now I can see the sources. Rarely do I click on an actual feed…mostly I walk through folder by folder and mark all as read unless something catches my eye. Oh - and the keyboard shortcuts are nice too. Eventually I’m sure I’ll want search capabilities, but hopefully that will be done before I actually need it. In the meantime, Google can work on the offline issues. ;-)

Software I install

Thu, 09 Aug 2007 13:49:00 -0700

I’ve been working on this list for some time, and now I think it’s finally near comprehensive. This is the full list of software I install. Software is organized by category, where:

Basic software gets installed on any machine
Home software gets installed at home
Development software gets installed on any machine whenever I need it
Business software gets installed on my work laptop
Server software is installed on my home server (expected to be up 24x7)
Portable software runs directly from my USB key
Portable - Development software is used for development from my USB key

Make sure to check out the other sheets in the spreadsheet. They include:

Resources to find good tools
Firefox addins (there’s no good way I know of to simply install a large number of addins, which would be nice for initial install)
Configuration Options, which describes how I configure the various OS and application options (it would be nice to optimize this). I don’t have that list built yet, but hope to soon.
Software I’ve used in the past and have removed for one reason or another. Interesting trip down memory lane for me, but I thought keeping the list would be good for others who might want to avoid some of the items on that list

I’ve also setup (but not tested) two InstallPad files to help streamline install of several of these applications. These are items from my Basic category and items from the Home and Development categories.

I’ll keep the list updated and current, and when I do an initial install, I’m sure the Installpad files will get tested and the list updated with stuff I’ve missed. Here’s the URL again for reference:

http://spreadsheets.google.com/pub?key=p9VpnOc6eSLLWVL_yUTWrFg

The things you learn about yourself

Wed, 08 Aug 2007 09:41:00 -0700

I can’t believe it took this long to find out this little factoid about myself. I was on the phone today with a customer service agent, and I announced my name as I always do: “Hi, this is Emil Lerch. I’m calling because…”.

In this case, however, after I described the issue I was having, the customer service agent said, “I’m sorry, did you say you were with Merrill Lynch?”. Thinking about it, I can totally see how under a bit of distortion, my name does sound a bit like a giant brokerage firm. Who knew?

Really, really cool web site

Mon, 06 Aug 2007 16:36:00 -0700

Lately I’ve been questioning the value of Quicken, since the integration with financial institutions is finicky and getting everything updated appropriately takes a really long time. Typically, I use Quicken for 3 purposes:

See our net worth
See how we spend our money
Seed tax software with donations and such

A distant 4th is to see how our investments are allocated and determine if any rebalancing needs to be done, but the last time I did that, we just put everything into a spreadsheet.

In order to accomplish these goals, usually once or twice a year I spend 2-3 days getting everything entered into Quicken and reconciled. I don’t really think we’re that complex, but with 2 Roth IRAs, 1 regular IRA, and investment account, 2 individual bank accounts, 1 joint bank account, 2 401(k) accounts plus 2 profit sharing accounts, 2 stock option accounts, 1 529, 2 credit cards, 1 investment account for Kathryn and a bank account for Kathryn, we have over 18 accounts (there are a few others…I thought one time I counted 23, but since then there’s been some consolidation), Introduce dependencies between accounts (e.g. you should reconcile credit cards before the bank statements), and things get really messy.

All this leads to today, where I thought there must be a way to accomplish my goals faster…I’m a data packrat and it rubs me the wrong way to abandon the long history of data I have in Quicken, but I really don’t use all that stuff. I did some Google searching and ran across this marketplace interview that describes a website called Yodlee MoneyCentral.

This site is way, way cool. I set up most of my accounts in about an hour, mostly because I already have my account login information stored and synced up in KeePass Password Safe. It keeps track of all my financials (mortgage, house pricing manual or through Zillow, credit cards, bank accounts, investments) as well as rewards accounts (airlines and such), and provides automatic transaction download and categorization.

Other than entering in usernames/passwords (yes, this can be a leap of faith), the site does the rest, and seconds after setting up the accounts you can see how your spending is categorized, your total net worth, portfolio positions, etc. Now that (almost) everything is setup, I’m interested to see the charting available for history…it lets you see how your investments, transactions, and net worth change over time, but it can’t get a large amount of historical data on initial setup (I suspect this is due to the limitations of the financial institutions more than Yodlee). It also provides email alerts for bills, balances (high or low), large transactions, securities expiration, airline mile postings/expirations, etc.

So, the information for this month is available now, after a one-time cost of about an hour. Had I tried to see total net worth in Quicken today, I couldn’t, because it would take me about 16 man-hours to bring it up to date. There is still an account or two that it was having problems with (I think this is due to recent web site changes at TD Ameritrade), but already I know my net worth +/- 10 percent (which would be reduced to 0 if I wanted to login to TD Ameritrade and add the summary amount to the total on Yodlee).

Even better, the figure it reported was more than I thought. ;-)

Good (and cheap) computer

Fri, 03 Aug 2007 16:28:00 -0700

So I’ve been running this machine now for the past few weeks, and so far, it seems great. Most parts were purchased from newegg, and here’s the final list of parts purchased. In all cases, I went for maximizing price/performance ratio. The case in particular was a great win…I was amazed how good it was for the price.

Here’s the bill of materials (note that the CPU was purchased from Intel rather than newegg, and parts that no longer are sold by newegg don’t show up on the wishlist anymore): http://secure.newegg.com/NewVersion/wishlist/PublicWishDetail.asp?WishListNumber=5160671

Here’s a quick rundown:

Rosewill R103A case: $25
MSI NX8600GT Video card: $112. This is a great card, but actually my only real mistake. When I ordered the motherboard, I went with a 965P chipset, thinking it had integrated graphics. It did not. The downside was a little extra cost. The upside is that while I’m not a gamer, this video card will support gaming graphics, and has dual monitor support as well.
Gigabyte GA-965P-DS3 motherboard: $120. Looks like they aren’t making it anymore, but it’s pretty good
4GB (2 modules of 2GB each) GSkill memory: $244. This is designed for dual channel operation, and one of the places I loaded up, as I run multiple virtual machines on the computer. The project actually consolidated 2 machines onto 1…I’m probably one of the few consumers in the world doing server consolidation in their house.
Scythe CPU cooler: $50. Not needed, as the CPU comes with its own fan, but this is quiet. Between the fanless video card, quieter hard drive and this cooler, I can just barely hear the machine.
Western Digital 120GB SATA hard drive (OEM): $50. Again, not exactly needed as I had a hard drive in the other machine, but quiet and fast.
Core 2 Duo E6600: $123 with Intel employee discount. Here I really saved some money, but had I not gotten the discount, I would have gotten the E4500, giving me VT for $146.

Reused from one of the old machines was a DVD drive (and keyboard/mouse/monitor), but had I needed to purchase, I would have gotten the Lite-on LH-20A1L-05 SATA burner with lightscribe support for $36.

This gives me a very nice machine at $724 total cost. Had I not made my motherboard error, it would have cost $612, and if I were doing this for someone more normal, it would have cost $526 by adding the DVD burner and removing 2GB of RAM.

Interestingly, Windows Vista reports an experience of 5.2, due to the processor…

How many popups does it take to get your health benefits?

Thu, 02 Aug 2007 10:59:00 -0700

I find that internal sites and “logistics” sites (e.g. travel) tend to have a crazy number of popups for no reason. Seriously, don’t the people at these companies use the Internet? There is a place an time for a popup, but these guys go nuts. Here’s the popup count for a typical “view my claim status” on Intel’s (outsourced) health benefits site:

Go to the home page. Browser reports that a popup was blocked. Whatever this is, it’s irrelevant to what I’m doing. I’ve never viewed this popup.
Click login. A popup window appears to collect my user/password. The original screen is not used again.
Click “manage my account”. Another popup. At least in this case, the old screen is marginally useful, as I could go back to it to get to another transaction (like changing my address vs. managing claims, where I am now). Of course, I’ve never gone in to do more than one thing at a time, so leaving this window is useless to me. I’m now at 3 pages open and haven’t even gotten to my claim yet.
Click my account (flexible spending account), and claims, without popups. Awesome. These are 2 out of 4 mouse clicks on the site that did not generate popups.
Click the claim to see details. Another (blocked) popup, but this one actually is important and the only popup that I might make an argument to keep, as it is intended for very detailed information, has a 2D barcode and is generally printed out, then closed.

For those that are counting, I now have 4 windows open, and generally, I will never do anything except print the last window and close everything. My absolute favorite has to be the popup on clicking the login button…there are more things wrong with that than I can even count.

Tip for central air conditioning

Wed, 01 Aug 2007 09:57:00 -0700

A little bit of randomness. We had an air conditioning tune up yesterday. Basically, they just checked everything out…no actual maintenance was required). During the check-out, they asked if I had a bunch of vents closed. I had been closing the downstairs vents to force the cold air upstairs in the summer. However, the idea backfired somewhat, as this just created back pressure that kept the cold air sitting in the vents rather than coming through the house.

The technician told me never to close more than 1 or 2 vents in the house at any given time. Another AC tip - to reduce the temperature difference between upstairs and downstairs (without zoning), set the fan to “ON” rather than auto. The AC compressor won’t kick in unless the temp gets to high, but leaving the fan on will ensure that the air continues to get circulated throughout the house.

When bad decisions collide

Thu, 26 Jul 2007 13:08:00 -0700

Sometimes this stuff drives me nuts…I get an email with a link, but the URL itself is attached to a graphical button, and Outlook doesn’t let me copy the hyperlink off an image. I click the button, which launches my default browser (firefox), which the site doesn’t support, so they redirect me to URL/unsupportedbrowser.aspx in a way that doesn’t allow me to hit the back button to see the original URL.

In fact, the application, which literally includes a button that adds you to a list of people attending a meeting, does NOT actually support firefox. I have to assume the time they spent coding in a nice “your browser is unsupported” message far outweighed the time it would have taken to support a firefox user clicking a single button, but that’s another gripe session. For some reason, this particular app also crashes firefox when I add the site to the IE Tab Sites filter and let IE run the app.

My easiest method for dealing with this situation (without changing default browsers just for this scenario), is to change my firefox user agent, let the broken application load, and copy/paste the URL into a real IE.

Not until crazy stuff like this is resolved will average users really pick up firefox. Well, that and getting the browser pre-installed on machines. ;-)

D-Link DIR-655 gets certified 802.11n draft status

Mon, 09 Jul 2007 10:26:00 -0700

I do like this router, and it’s good to see it come through certification. It seems like we’ve been waiting forever for 802.11n finalization, but things are moving slowly.

Review: http://www.smallnetbuilder.com/content/view/30087/96/
Product Page: http://www.dlink.com/products/?pid=530&sec=0
Certification press release: http://www.dlink.com/press/pr/?prid=341

NIH Syndrome: symptom or cause?

Mon, 09 Jul 2007 08:20:00 -0700

I was listening to a podcast this morning and the person was talking about NIH Syndrome. This is one of those things that consistently rubs me the wrong way, primarily because I believe that managers’ actions and managers’ talk diverge significantly in this area. Many things that are not invented here cost money. Because, as a coworker once put it, “man-hours are free”, the tendency for a lot of managers is to use their people to re-invent the wheel rather than putting together whatever justification the organizational bureaucracy needs to cut a PO. Since people are now re-inventing the wheel, to satisfy their ego they must be able to do it better. Do this enough times, and it’s clear that NIH really is the symptom of poorly performing management.

Of course, person-hours are not free, rather they’ve already been negotiated for, while budgeting extra dollars for outside products or services often involves more administrative work. It’s the management path of least resistance.

Why, oh why

Fri, 22 Jun 2007 14:23:00 -0700

Do I get this page, telling me that I can’t get FiOS, when not only is it possible for me to get FiOS, but I’m connecting to the web site over a FiOS connection!

You’d think they would have fixed this in the past 18 months, but I guess they’re not interested in new customers.

(I go to the site periodically to find out if I can add TV to my existing FiOS, but I’m not sure why I bother…I apparently need to call to find out. When I originally had FiOS installed, the site said I couldn’t get it, but customer service looked in their computers and found out it was no problem at all)

Smartphone application update

Wed, 20 Jun 2007 17:47:00 -0700

I had some time the other day so I actually looked at the stats for the 2 smartphone applications I wrote a while ago. I was shocked to find that Notepad had 558 downloads and Stopwatch had 275 downloads. I don’t know how many people are actually installing and using these applications, but I’m glad I could help make their lives a bit easier.

Excellent illustration of a common business problem

Mon, 18 Jun 2007 10:08:00 -0700

Even one presentation like this video is too much. Tell me how you plan to achieve actual business results, and give me some realistic numbers. The hockey stick doesn’t cut it without significant justification. What really bothers me is when I’m told (especially by my own management), “the numbers are just too hard to get, but there is absolutely a positive ROI to this”. Nine times out of 10 it’s a cop out, and the translation is that you’re just too lazy to do the analysis you should be doing.

Home network

Fri, 15 Jun 2007 13:37:00 -0700

In addition to all the recent house changes, I’ve been reconfiguring the home network to improve a couple things, and it’s been quite the saga. Here’s what I wanted to fix:

Move off of WEP security in favor of WPA2. This is not just for better security, but because, at least for my router, you cannot use WEP if you want to support 802.11n clients. I don’t have any yet, but don’t want to be prevented from the faster speeds, either.
Provide wireless coverage in the kitchen nook. This is the only area of the house I ever have problems with, and while I don’t sit there and use a computer, anyone we have over tends to do just that.
Provide a stronger wireless connection for the TiVo, which would stop the problem of my mp3’s pausing during playback and really, really slow transfers of shows from computer to TiVo.
Spend no money

One slight problem with #4 is that the wireless adapter I use is not the official Tivo Wireless Adapter (which didn’t exist a couple years ago). TiVo doesn’t support WPA in adapters other than their official one, so it created a problem.

Well, I had gotten an offer from Fon to “upgrade” my Fon router to a La Fonera for free. I originally signed up in order to get the very nice Linksys WRT54GL, and was thinking about repurposing it once fulling my obligation was over, so this was an opportunity to repurpose the Linksys early.

Unfortunately, what I needed from the Linksys was to create a wireless to wireless bridge. This would connect to the 802.11n/g/b network with WPA2 as a client (direct wired not an option if I were going to solve problems 2 and 3). It would also work as an AP broadcasting on 802.11g/b with WEP to let the TiVo work. I took a look at a wireless distribution system to do this, but all settings must be shared, and I needed to have different security.

DD-WRT did not allow this setup, but I was pointed out to a beta version of v24 that has this bridging feature, which they call the Universal Wireless Repeater. The beta version worked, but in order to get traffic from my main router to route to the Linksys, I needed a static route to be configured. This feature only became available for my DLink router a few weeks before, so I went through the upgrade and I was finally all set up.

In testing, I found one more problem. The protocol TiVo uses for transfer of TV shows (but not anything else) requires the TiVo to be on the same subnet as the server. Bummer. Everything else works great, and at this point I’m resigned to a second transfer (server->Laptop->TiVo) to accomplish that.

As an aside, although I love my TiVo, unless something changes, I’ll be forced to move on at some point, based partly on issues like this, and partly on the lack of HD support without shelling out a ton of money on a series 3 and paying a monthly subscription fee (I have a lifetime subscription). While I think that Media Center is complex, expensive, and unstable for consumers, it’s likely HD support will be cheaper unless TiVo lowers prices, and I’m sure things will get more stable in Media Center world as we move forward.

Yikes…long post. Anyway, here’s a picture of where I ended up.

Aaaahhhhh....

Thu, 14 Jun 2007 16:17:00 -0700

Kelly and I have been on a tear lately (as much of a tear as us analytical/methodical people can be) getting the house the way we want it. We’ve been selling some stuff on craigslist and making a ton of purchases. Here’s a quick list of what I can remember doing over the last 6 months:

Had house interior painted (this is huge, considering our two-story formal living room)
Created a “photo wall”
Purchased three bookcases, gave one bookcase to goodwill
Moved some picture frames to a new wall
Got an entertainment center to replace our old one (known as “Pac Man” after we stuck our new plasma TV into the space designed for a regular CRT)
Sold floor standing Infinity speakers, center speaker, and rear speakers, purchasing a full set of Polk speakers
(in process) Selling coffee table and end table, with the intension of purchasing “cubes” for the family room

Well, today I got a hidden IR system, which so far is making me very happy. The problem we ran into after all the entertainment center business is that the center speaker, which clearly belongs in the center, would cover the IR sensor on the TV, rendering remotes useless from one half of the room (the half we use the most). The only way to alleviate the issue was to move the center speaker all the way to one edge of the TV or the other, which no longer placed it in the center. The system has completely alleviated that problem. I’ve only used it in tests so far, but it seems great.

Password safe and folder share

Wed, 13 Jun 2007 10:04:00 -0700

I recently installed both password safe and foldershare, with the intention of using them together to:

Maintain my passwords in a more secure way (password safe)
Have a copy of the passwords synced on the computers I use (foldershare)

The solution works great, and is highly recommended. Foldershare is blocked at work, but my password file doesn’t get changed too frequently, so I’m ok with it. Prior to finding foldershare, I actually started coding my own solution - this was a great find, and I can’t wait to see what Microsoft does with it post-acquisition.

Cookies and affirmative action

Tue, 05 Jun 2007 05:30:00 -0700

Yesterday I heard a very strange analogy. There is some political unrest right now in northern India, and it seems as though the cause of the unrest boils down to a certain class of citizens that want to be included into the country’s version of affirmative action. The affirmative action system was established when the country was founded in order to end the caste system, and was originally intended to be ended 10 years.

This discussion happened a few minutes after we were talking about how Intel acquired a group of people in India who were used to free cookies, and the near revolt they had on their hands when they tried (unsuccessfully) to end the perk. They got there eventually, but not before having to end the lease on the building.

On explaining the current events, the comment was made that it’s actually pretty similar to the cookie situation…once you give people something, they want more, and don’t dare take it away…

PS: I couldn’t find a good reference point for this URL, but I thought it might be interesting for my reader(s) to have more background on the caste system in India.

4AM in India again...

Mon, 04 Jun 2007 16:18:00 -0700

I don’t know what it is about my body clock, but very consistently, when I travel to India from the west coast, I get crazy awake at 4AM for probably the first week. I arrived in India Sunday around midnight (that’s about noon west coast time). Now it’s Tuesday morning at 4AM (Monday afternoon about 4PM west coast time) and I’m wide awake.

My strategy here usually involves allowing myself some awake time…usually about an hour, and then going back to sleep. It may not be the best strategy, but usually I find that trying to get back to sleep immediately is pointless anyway, so my real alternatives are to either indulge myself with some awake time or toss and turn for an hour trying to get back to sleep.

I’m only here for a week this time, so my body will probably finish adjusting just when I’m ready to go home.

Coolest tip of the day - for Gmail

Thu, 31 May 2007 15:20:00 -0700

I read this piece on lifehack.org today, and saw this killer tip:

Send spam to trash. Instead of having Gmail-filtered spam go into your Spam folder (and have the annoying count of unread spam by the folder’s name), set up a filter with “is:spam” in the “has the words” field (just click “OK” on Gmail’s warning dialog box when you click next step) and “Delete it” as the action. Now all spam messages will go in your trash.

How to write a question that answers itself

Thu, 31 May 2007 09:36:00 -0700

Just ask a yes/no question in a FAQ about whether or not a feature exists. The answer to that question is clearly “Yes”. For instance, this question from Google calendar’s FAQ:

Can I check my calendar information via SMS?

Clearly I can check my calendar information via SMS. I don’t see another question saying “Can I check my calendar information via carrier pigeon?”. The reason I don’t see this question, is that I can not check my calendar information that way, and Google isn’t typically going to put a FAQ item out there to tell people what they won’t do…FAQs are as much of a marketing tool as a way to disseminate information.

So the answer is yes, the real question is how would I do that? Click the link and find out. Kind of makes you wonder why they don’t rephrase the question as “How do I check my calendar information via SMS?”

Popular baby names

Tue, 29 May 2007 08:28:00 -0700

According to
popular baby names from around the world, my name has ranked #7 in Finland this year and #10 in Sweden. As my name is definitely not known very well in the US, it’s good to see this kind of news from time to time.

Spam

Tue, 29 May 2007 08:00:00 -0700

I’ve had now 3 different fraud attempts in the last week (we need your help to claim x million dollars…) appear in my inbox, both at work and at my personal address, making it past all spam filters. I have another 2 in my junk email at work, where I hardly get any spam at all. It’s odd to me that this has happened so suddenly. Maybe these people are in the US and have more time over memorial day.

In other spam news, I was going through my spam the other day to make sure nothing important was being filtered out (I actually found that the filters were grabbing my Vonage email, so it was a worthwhile exercise). Most of the spam was related to certain drugs, and it made me wonder what caused these drugs to be responsible for 80% of my email. This is my conclusion:

Viagra/Cialis/etc. are marketed to help with issues that people might not want to discuss out in the open - email gives them anonymity and frees them from embarassment.
They are expensive, and those that want the drug have a strong desire to realize some cost savings. This thought is reinforced by the fact that a good portion of email not relating to these drugs are referring to ways of getting cheap drugs in general.
They are expensive, meaning that the spammers realize a lot of profit if they can sell fraudulent (or maybe even real?) pills.

Next in the list was spam related to attempts to gain access to my personal information (phishing), followed by stock tips. Stock tips have historically been the most successful at getting through the filters. They tend to also use images containing text, rather than just sending their message as text in the body. I don’t usually look at any spam that sits in the spam filter, so it’s unclear to me how many other messages use the technique.

Quick tip on google calculator

Mon, 21 May 2007 09:36:00 -0700

I noticed a while ago that google toolbar in both firefox and IE has a very nice feature. As you type a search, it will do an autocomplete based on popular searches. However, if you put in google calculator terms, it will also provide the calculator answer in the dropdown, so you don’t even have to hit enter to see the answer. I used this yesterday, for instance, to determine how many teaspoons it takes to make 1/2 a tablespoon (1/2 tblsp in tsp).

For the love of god, just do it...

Fri, 18 May 2007 12:04:00 -0700

Sometimes it’s the little things that just drive you nuts. Like all the meeting requests I get from people with location “TBD”. At Intel it takes about 1 minute to schedule a conference room or phone meeting. You’re already spending at least 2 or 3 minutes finding the right attendees and putting together an agenda for the meeting, why postpone this minor inconvenience? Inevitably this “TBD” gets sent in another update about 15 minutes before the meeting. Spend a little bit of extra time and save yourself the hassle of:

People who printed out or downloaded meeting information and don’t have network connectivity just before the meeting
Forgetting about to fill in the TBD completely and having to start the meeting 5 minutes late as you scramble to schedule something (1 minute because you’re late to begin with, 4 minutes trying to go through a 1 minute procedure while people are IMing you to tell you that you forgot to add phone conference information)
Not getting the information to people who have been forwarded the request

Getting around lousy software

Tue, 15 May 2007 11:21:00 -0700

I like HP’s printers, but absolutely hate their software. Here’s a list of problems I’ve had with them off the top of my head:

Software assumes you can connect to the printer at all times - if you’re not, periodic episodes of 100% CPU utilization occurs. Disabling WIA (Image Acquisition Service) cured that problem.
Software periodically crashes for no particular reason, and when not using the software actively. I believe this is related to the “always connected” issue above.
Software works about 50% of the time when scanning.
Software is extremely slow to shutdown and sometimes refuses to do so
Software blue screens the machine
Software that likes to update itself but usually can’t connect either due to firewalls (despite proper proxy config) or some other reason
Software allows copying data off a memory card inserted into the printer, but does not copy/set a create date on the files, which crashes lots of other applications

I haven’t seen the last problem for a while, and a higher % of the issues I see seem to revolve around scanning functionality. I’ve been trying to document a way to consistently scan without spending an hour trying to do it, and I’ve finally found the way…avoid the software as much as possible. Here’s my current method:

Insert memory card into printer
Scan document and send the scan to the memory card
Copy the files from the memory card to the computer
Use a “touch” utility to set the create date on the file to avoid the last problem above

As a software guy, this stuff probably bothers me more than most people. I can’t imagine trying to work around all these issues if I were a normal person - I’d probably try to scan once or twice and then return the product. I understand the need for HP to ship early and ship often, but you need a complete product…not 75% of one (hardware and half-working software).

Not all Wal-Marts are created equal

Fri, 11 May 2007 10:46:00 -0700

I recently used Wal-Mart’s Site to Store (not a bad service!) to buy an entertainment center for our house. The particular store I had it delivered to, however, was ickier than most Wal-Marts. Allow me to illustrate this store’s placement on a standard statistical bell-curve of Icky-ness:

Picopad - seems cool so far

Thu, 10 May 2007 13:59:00 -0700

I just got my Picopad in the mail and slipped it into my wallet. Everything seems as advertised so far, but I have yet to try it out in a live environment.

I’ve loved my Signature knife for a long time, but I’ve had a problem with using it while traveling. I don’t like to take it off my keyring and check the knife, so I’m stuck leaving it at home, and it’s when I travel that I seem to need it most. 90% of the time I pull the knife out is to use the pen that it has. The picopad solves this problem, and gives me something to write on as well. When not traveling, I’ll almost definitely keep my knife with me, but now it’s not so critical.

Very nice trend

Tue, 08 May 2007 08:55:00 -0700

I like the idea of being able to quickly have a full OS and environment available for use through the distribution of files for a virtual machine. Especially with the cost of VM software driving down quickly (for personal use without support this is now usually free), this is especially useful for development and testing.

A couple quick links on this:

rBuilder Online - Release: LAMP appliance (I found the latest release to be broken, but this one works - your results may vary)
Orcas Community Technology Preview. Next generation Visual Studio on a virtual machine - no need to risk pre-production software on a machine you care about.
Windows XP with IE6 or IE7. Great tool for testing
Directory of user-contributed VMWare images for a variety of purposes

Also there are some conversion tools for those that prefer VPC over VMWare or vice-versa:

I’m sure other converters are available for Xen, etc. As other food for thought, many virtual hard disks could fit on a USB key, making these OS’s fairly portable.

Dual monitor setup

Fri, 27 Apr 2007 13:06:00 -0700

I just changed offices and so I’ve taken the opportunity to try out a new monitor setup. My primary machine is a laptop and I have a external monitor, so I’ve always enjoyed the extra real estate of an extended desktop, at least when I’m in the office. However, I was finding the setup of limited use the way I had it, with my monitor above the laptop. After moving to the new office, I put the monitor to the (left) side of the notebook, and now on day 2 I’m finding it to be very comfortable and useful. Biggest downside so far is that it’s on the left side, so I almost expect it to be the primary monitor (I assume this is my cultural expectation since I read left to right). I’d rather have it on the left for other reasons, and I don’t want it to be considered the primary to make it easier to dock/undock the laptop without running.

Just a quick Friday rambling…

Back from Mexico - that's a big fish!

Tue, 24 Apr 2007 14:57:00 -0700

Here is the head of the fish I caught in Mexico. It’s a Roosterfish. We had a great time hanging out and pretty much lounging, but on Tuesday we went fishing and I taught the locals a thing or two. This fish is 63" long and 96lbs. The world record Roosterfish is 66" long and 114lbs. The fish my brother-in-law caught was no slacker either. We hooked into these fish within a few seconds of each other, and his fish was 54" long and 83lbs. In the picture below, we’re a little hunched over because that is really heavy, and slippery.

Blogger and Outlook 2007

Thu, 12 Apr 2007 08:17:00 -0700

I noticed that atom feeds from blogger in Outlook 2007 appear to be broken. All posts (at least on my machine) show up with a date of 12/31/2006. This help file shows you how to get to an RSS feed, which works correctly in Outlook. Note that the trick does not work with the old style blog URL (http://example.blogspot.com/atom.xml), but only with the new style URL available only with the new blogger (http://example.blogspot.com/feeds/posts/full?alt=rss).

This post helped me figure out the new URL. Unfortunately, all I’ve seen in blogger help is how to embed the URL in your template rather than a help file telling you what the URL actually is. For reference, here are the URLs for this blog, which I’ll add to the template shortly:

Atom 1.0: http://emilsblog.lerch.org/feeds/posts/full
RSS 2.0: http://emilsblog.lerch.org/feeds/posts/full?alt=rss

Passport saga

Thu, 12 Apr 2007 08:07:00 -0700

We are taking a family vacation to Casa Manana in Bucerias, Mexico, just north of Puerto Vallarta. We originally applied for Kathryn’s passport back on January 18th, and were given March 15th as the date for the passport to get here in the mail. Due to the recent changes in travel regulations, we expected it would take longer, but we decided 3 months (12 weeks) was plenty of time, and we did not need expedited service. The web site currently claims processing times of 10 weeks, although I believe at the time it was saying 8 weeks.

March 15th flew by, so we checked the status online around March 29th. The site said that we’d get the passport April 9th. This was pretty close to our April 15th departure, so I called on April 3rd to find out if this was a problem, and if so, what I could do about it. When I called, I literally got “all circuits are busy, please try again later”. A redial later, I got through, and the phone menu starts with, “We are currently experiencing unprecedented demand for US Passports. We have added staff and are working overtime to make sure that no travel plans are effected. If you are not traveling in the next two weeks, PLEASE, hang up. (then something to the effect of “it will be ok”)”. Option 1 and option 3 later, I’m told, “all our agents are busy, we can not answer your call at this time”.

Several more redials later, and I actually get into the queue. I wait a relatively short 30 minutes and the agent tells me, “oh, you’re in good shape, this is in book print”. I asked what that meant, and they said that it stays in book print for a day and then someone physically compares the passport to the application, a process that also takes a day. Then it gets shipped out priority mail (2-3 days). The woman told me that with Easter and everything, and “you know how the government is” (I love that coming from a government employee), that I should see it Saturday or Monday.

Monday goes by, and no passport. Tuesday goes by, and no passport. So I check the status online after the mail came, and now it says that it shipped, but it’s estimated to arrive “on or about 4/14/07”. Umm…we leave on 4/15, so what do they mean with that “about”? So, I give them another call on Tuesday (April 10th…5 days before we fly). After the prerequisite redialing and 30 minute wait, I get through and they said I should be in good shape. It physically was on a truck on April 9th, so I should see it Thursday or Friday (April 12th or 13th).

The story does have a good ending, as the passport has arrived today (April 11th). Talk about cutting it close, though…

As a quick postscript to a long post, Kathryn got one of the new RFID passports, and it’s really fancy. I think they did a very good job with the design and artwork, and I can only assume it’s less subject to counterfeiting. I’m not sure how well the face recognition will work with Kathryn when she’s 4 years old, though.

Calendaring events way in the future

Wed, 11 Apr 2007 16:07:00 -0700

So after receiving Kathryn’s passport, I had to deal with the fact that it will expire, and that I have to remember to renew the passport when it does. I believe that software definitely has a half-life, so I had to question the use of Google calendar, which for our family is at calendar.lerch.org through the Google apps for your domain service.

While I like the service, will it still be what I’m using when my passport expires in 2014? What’s the alternative reminder system (on or offline) that I can expect to be using at that time? In the end I decided that if not using Google calendar I would be migrating the data to whatever would be in use at that time. Since reminders work via SMS, I’m also assuming that the reminder system migrates to whatever is in use.

I guess the fall back is to just check the passports before travel, but the general issue still exists. Given the transient nature of systems (software or process), what assumptions can you make to ensure that you get when you expect in 1 year, 5 years, or 10 years?

Mappy Hour

Wed, 11 Apr 2007 09:00:00 -0700

I was just forwarded a link to the cool site of the day. It’s a mashup combining happy hour information with Google maps. Awesome.

Blog URL change

Fri, 06 Apr 2007 08:59:00 -0700

The current URL will still work, but I’ve set the blog up so it can now be accessed via emilsblog.lerch.org.

Our family blog is at blog.lerch.org.

Coolest thing I've seen all day

Thu, 05 Apr 2007 16:28:00 -0700

Google maps launched a way to annotate and save your own custom maps. I’ve started a map of my overall travels, and one specifically focused on places I’ve been in India. I see this being a big project, but it would be nice to have URLs to pictures, etc. at each site.

More tangible value for this can be seen in the map I’ve done to show where to park and how to get to Bethany Community Network meetings.

Cool.

SubSonic - The Zero Code DAL

Fri, 23 Mar 2007 16:18:00 -0700

If you’re a .NET person with ruby on rails envy, check out SubSonic - The Zero Code DAL

Holy crap that's a good router

Wed, 21 Mar 2007 10:24:00 -0700

So my router problems came back mysteriously, and disappeared just as mysteriously several days later. I’m still not sure what the problem was, but in the meantime I decided I had spent enough of my time trying to diagnose the problem and bought a DLink DIR-655. The router still requires a lot of reboots during configuration, but not nearly as much as my original one. However, this is rock solid, and fast. I couldn’t believe my eyes while I’m on the phone (using VoIP through the Internet) and watching a 435MB download (Visual Studio SP1) clicking off about 1MB/sec, which is nearly my rated download speed through Verizon FIOS.

Oh, and I was also connected to work at the time. Also, I haven’t had a single unscheduled reboot since setting it up.

Membership and Role providers for MySQL

Wed, 21 Mar 2007 10:15:00 -0700

Stumbled on this the other day while looking for a way to use MySQL with ASP.NET and thought I’d share.

Membership and Role providers for MySQL - The Code Project - ASP.NET

Is all press good press?

Mon, 19 Mar 2007 12:37:00 -0700

I just noticed this post, and it’s interesting to me that a company’s internal OS upgrade schedule would be taken so seriously, but this is Intel, and that, I guess, makes things a little different.

New Role

Fri, 16 Mar 2007 10:42:00 -0700

So, I’m all set…getting settled into a new role as a capability manager at Intel, working on non-Direct customer applications and data. If that’s clear to you, let me know because I have some questions myself. ;-)

It’s been quite the start - this is actually a very high priority in our group right now due to the variety of systems that contain unique sets of customer data. Everything is coming to a head right now due to a desire to converge onto a single platform, so my first program and project will be on the critical path to at least 5 other programs. I’ve been in 3 days of meetings in my new role already this week (while trying to carve time out for transition out of my old role), and I only knew what I’d be doing Monday afternoon. Yuck. Anyway, enough work stuff…

Cool site of the day: http://becomeanmm.com

Wed, 14 Mar 2007 14:58:00 -0700

ok - this is cool. Thanks Matt!

Here’s a picture of me from the site:

Role changes

Fri, 09 Mar 2007 14:31:00 -0800

It appears as though my time in my current position (as a product manager for systems supporting our customer support processes) at Intel is about to end. The new adventure is still to be determined, but I’m rapidly closing in on 3 alternatives. I should have more information, but not necessarily a final answer, sometime next week.

It depends

Fri, 09 Mar 2007 14:30:00 -0800

I’ve heard the expression that the answer is almost always, “it depends”. Whether or not the answer is “it depends”, of course, depends on certain factors. ;-) Math problems usually have a concrete answer, for instance, at least until you get into more advanced subjects.

One thing I try to keep in mind, however, is that an expert in the field can always tell you what the answer to your question depends on. A novice will generally give you an answer without first understanding the background and context for your question, thereby ignoring critical factors that your answer depends upon, or simply tell you that the answer depends, but will not be able to articulate what it is that will help you come to a conclusion.

To separate experts from posers, ask a question and see if the person can tell you the factors that would influence the answer.

Virtualization bad for hardware sales?

Thu, 08 Mar 2007 14:56:00 -0800

I just read this blog, which points to this article, and I disagree. Virtualization is good for hardware sales, because it shortens the hardware refresh cycle. It will certainly cause a shift toward a fewer number of larger, more expensive servers, but it also means that replacing those servers with faster ones will be a lot simpler, and companies will be much more likely to throw more hardware at performance problems than they were in the past.

What good is a phone call if you can't speak?

Wed, 14 Feb 2007 14:34:00 -0800

So having a daughter is fantastic, but it’s been interesting to see how logistics work when you can’t sign your name…here are two recent logistics issues and how they needed to be handled:

Obtaining a passport:

Kathryn had to get a picture taken, and show up with both Kelly and I at the passport office with photos in hand, a birth certificate showing Kathryn was born in the US and naming Kelly and I as parents, and our drivers licenses to prove we actually are the parents
Once we get her passport, it is not valid until signed. The signature? One of us prints Kathryn’s name and then signs our own name, adding (MOTHER) or (FATHER) after the signature. Here’s the official FAQ item. Oh, and the requirements for obtaining a passport for minors.

2. Depositing a check (that was addressed payable to child):

Account needs to be opened as UTMA (or UGMA if you’re from a few states that don’t recognize UTMA yet). Parent is custodian.
Check needs to be signed by me as a custodian, payable to the bank, for benefit of the child. For Kathryn, the full check endorsement went something like this: Custodian for Kathryn payable to / FBO Kathryn

Also fun is the fact that certain things require an email address. Kathryn’s email address is established now, but it will be several years before she reads her own email…

Light reading

Tue, 13 Feb 2007 11:30:00 -0800

On the way back from India I read an article (PDF) on tax diversification when working on retirement savings planning. I thought it was an exceptional article…absolutely recommended.

Driving in India

Mon, 05 Feb 2007 21:43:00 -0800

Unless you’re native, don’t do it. I just uploaded a few short videos of the driving experience this past weekend in Delhi and in Agra. I’ve been staying in Bangalore, but the driving experience is pretty much the same. I also have a very short (but boring - buildings don’t do a lot of stuff) video of visiting the Taj Mahal.

Trip to the Taj Mahal/Fatehpur Sikri

Mon, 05 Feb 2007 02:41:00 -0800

So I just finished uploading pictures of my trip to the Taj Mahal and Fatehpur Sikri. Fatehpur Sikri was very impressive, partly because I had no idea what to expect when I got there (by contrast, most people have an idea of what the Taj looks like and what it’s about).

The wikipedia entry gives some background on the site, but doesn’t mention a very important fact: I’m told that Akbar had 3 wives, one Christian, one Hindu, and one Muslim. The entire architecture of the place incorporates themes from all three religions. An example - the gate called Buland Darwaza is the same height as the Obelisk in St. Peter’s Basilica as an homage to Christians. The wikipedia entries even confirm this. Some of the pictures I took show the various styles. Here’s another example of what I’m talking about.

I’m also working on getting a translation for this Arabic plaque. It’s artistic in nature so it’s not easy to read, even for natives (I have a coworker who is Egyptian).

Yikes - long time

Mon, 05 Feb 2007 02:39:00 -0800

So I’ve totally gone dark. Short summary of what I’ve been up to:

Crazy work stuff (December)
Christmas with wife/daughter in Chicago
Crazy work stuff (January)
Business trip to India

I’m India right now. Will try to post more…

It's not surprising at all that people search for Yahoo

Thu, 30 Nov 2006 11:06:00 -0800

I couldn’t agree more. Very similar concepts to the pit of success, which I have posted outside my cube.

The Old New Thing : It’s not surprising at all that people search for Yahoo

Interesting conclusion...where did that come from?

Wed, 15 Nov 2006 11:08:00 -0800

I was surprised to see a stock I own (INPC) on this list today:
Russell 3000’s Best and Worst Performing Stocks: Financial News - Yahoo! Finance

It says YTD performance for INPC is 279%. I’m not sure how they get that…the closest that I find is a 140% gain if you bought at the 52 week low (5.02) and sold at the 52 week high (12.07). I don’t want to call them a liar or anything, but if YTD is defined as Jan 1 (8.76) to today(11.38), that is a 30% gain. If you bought when I did, you’re still down about 20% but a lot happier than when the stock was at $5/share.

IE7 Failed Experiment

Mon, 13 Nov 2006 09:58:00 -0800

With the recent production release of Internet Explorer 7, I thought it was time to revisit my default browser decision. I’ve been fairly happy with Firefox, but have had a few annoyances:

Copy/paste of HTML tables into applications (Microsoft Word and Excel) doesn’t work nearly as well as with IE
Proxy changes don’t happen without at least an extra trip to the options. To be fair, IE 6 didn’t handle it at all, but since the browser lacked tabs, starting a new instance wasn’t a difficulty
Some internal web sites designed the authentication in a manner that required another click to get in
When printing, some pages arrive with parts chopped off from the right
Some web sites (e.g. Wachovia) would break randomly for no apparent reason. Sometimes they would work, sometimes not

Well, I installed IE7 and it seemed ok. I found a mouse gesture plugin to compensate for the lack of something built in. I tried setting it as my default browser, but after using it a bit I had it crash a couple times. The session manager built in apparently assumes this will never happen, because not once did it restore my session from a crash, which is something that I’ve had for over a year in Firefox via the Tab Mix Plus extension.

Other than the fatal flaw, I agree wholeheartedly with the Cnet.com review of IE7. Firefox remains my default browser.

Not sure why I need this, but it's freakin' cool

Fri, 10 Nov 2006 15:35:00 -0800

So I went to my usual yearly eye exam the other day, and they had a new piece of equipment to help make sure that my retinas were healthy. It takes a picture of your retinas. Anyway, after the deal was done, the technician asked me if I wanted the photos emailed to me. Do I need that? Absolutely not. Is it cool? Yes. My response? Totally email it!

Without further ado…

Election results

Wed, 08 Nov 2006 09:02:00 -0800

Interesting day. Looks like the results are in, and democrats are taking control of at least the house. Call me cynical, but having different parties in control of different branches is my best case scenario. Gridlock is great…do we really want government messing things up more?

In Oregon, it looks like everything is going to the democrats, so I don’t get my gridlock. I guess you win some and lose some.

My favorite story of the day: Oregonians didn’t understand the assignment. Measures 46 and 47 in Oregon were dedicated to campaign finance reform, and were dependent on each other. Either they both needed to pass or nothing would happen. Well, measure 46 failed while measure 47 passed. The results weren’t even that close on either race.

Lastly, it looks like we’ll pass a $195M bond measure for local schools. Kathryn voiced her support of that!

SQL Server Hosting Toolkit

Tue, 07 Nov 2006 09:23:00 -0800

Just stumbled across this in my travels, and it looks pretty cool. This SQL Server Hosting Toolkit will generate database scripts from one database to deploy to another database, in an environment where full access is not available, say at GoDaddy.

Faking the soda machine

Mon, 06 Nov 2006 14:39:00 -0800

Quick tip to keep the amount of change in your pocket to a minimum. Suppose you only have a large bill in your pocket, and you want a soda (which of course is the only appropriate term for the substance). The soda machine will always stop accepting money after you’ve met or exceeded the amount of the highest price item.

The solution? Put in the change followed by your large bill. With any luck, you’ll get dollar coins rather than a mess of smaller change. For example, I paied for a $1.35 soda the other day with 35 cents followed by a $5 bill, resulting in 4 dollar coins rather than 3 dollar coins, 2 quarters, a dime and a nickel.

I, of course, think the typical soda machine user interface to be fatally flawed due to the inability of it to take more money than the highest price item (although I understand that the manufacturers don’t want the machines to become change machines). That debate we can have another time…for now, enjoy the workaround.

Small decisions that signal big ones

Thu, 02 Nov 2006 15:20:00 -0800

The other day I was visited by someone with a stack of manuals for a technology/product that we’re using in our group. I was asked whether we should throw those manuals out of keep them. I told them to go ahead and throw it out.

This is a technology/product that is currently in use. We’re moving to a new technology/product, and that small decision to throw out the manuals felt to me more like a massive strategic decision that yes, we’re committed to the new direction.

Bush keeps Rumsfeld, Cheney

Wed, 01 Nov 2006 13:50:00 -0800

Here’s something interesting I heard yesterday. Bush came out and announced he will hold onto Rusmfeld and Cheney until the end of his administration. Despite what you may think of the administration, it says a lot that Bush has to come out and make that kind of statement.

Sometimes you need to think through the ramifications

Wed, 01 Nov 2006 10:28:00 -0800

I’m currently on hold with my pediatrician and was amused by the phone announcement. After the usual “if this is an emergency please hang up and call 911” yada yada, it mentioned that “if you are a non-english speaker, please call us at xxx-xxx-xxxx”. Well, if I were a non-english speaker, would I really understand that request? The note was never repeated in Spanish (or any other language).

My Firefox tweaks

Mon, 30 Oct 2006 19:54:00 -0800

For speed, I’ve enabled pipelining. Here’s how:

about:config in the address bar
type “pipelining” in the filter
set “network.http.pipelining” to true
set “network.http.pipelining.maxrequests” to 8
set “network.http.proxy.pipelining” to true

For access at Intel, I also turned on http authentication:

set “network.automatic-ntlm-auth.trusted-uris” to “intel.com”

Optimizing Page Load Time

Mon, 30 Oct 2006 19:49:00 -0800

Stumbled across this page today…interesting discussion of how to bring down page load times.

Optimizing Page Load Time

Nice trick: Free screen captures embedded in powerpoint

Tue, 24 Oct 2006 13:47:00 -0700

Step 1: Install Wink
Step 2: Use how to embed a Flash file into Powerpoint to get the video in powerpoint

When good ideas go horribly wrong

Thu, 12 Oct 2006 13:13:00 -0700

I just signed up for a service to connect me to Intel’s network while out of town. I got 3 emails to confirm my service was active:

Email with my username
Email with my password
Email to state that my order was approved

Good idea…seperate the username email from the password email (although a better idea would be to allow me to choose username and password). However, email #3 included both my username and my password, so a) what’s the point of the other 2 emails, and b) the security advantage has been completely defeated.

Think people…think.

Tracking tool mashup

Wed, 11 Oct 2006 11:55:00 -0700

Here’s a neat little utility for tracking packages. I like the touch of adding RSS feeds. One small enhancement request…at least with UPS you get an expected delivery date. I assume many other companies provide that information, yet this site doesn’t give that information.

Why should it be this hard?

Wed, 11 Oct 2006 11:52:00 -0700

Very cool article on attaining the holy grail of synchronization, but why must it be that hard?

Couple security things

Wed, 11 Oct 2006 11:48:00 -0700

I found a cool post from Scott Hanselmann the other day regarding encrypted flash drives and password lists. I’m definitely using truecrypt, and I’m taking a look at password minder.

PDF files

Fri, 06 Oct 2006 10:19:00 -0700

Couple cool pieces of free sofware to do PDF stuff:

Foxit reader. If you’re tired of Adobe Reader’s slow load times, splash screens, intrusive update process and desire to install a bunch of other crap on your system, this is for you. My favorite feature (besides 1.5MB install and super-fast load time) is the ability to annotate PDF documents (highlight sections, strikeout, type onto forms that weren’t designed for forms, etc.). There are some nits with the software, but nothing worth going back to Adobe Reader for. Fantastic.
PrimoPDF. Installs a virtual printer that creates a PDF file. File->Print from any application and get a PDF output. There’s a few other pieces of software that do the same thing as well.

Foxit software also has a fairly cheap PDF editor even though you could technically edit using the reader and the PrimoPDF tool. This makes me wonder…is the one of the main benefit for PDFs (non-editable content without resorting to serious DRM technologies) starting to erode?

Source code and search

Thu, 05 Oct 2006 10:50:00 -0700

It was only yesterday I ran across a search engine for source code. Today, I find that Google is entering the same market. Cool.

Oh, and here’s the link to the actual Google search engine.

A few quick job links

Thu, 05 Oct 2006 08:50:00 -0700

Over the past few months I have thought quite a bit about resumes and jobs outside Intel. A few quickie links for those that are looking:

Seven deadly sins of resume design (lifehack.org, which is a pretty neat resource itself)
Software Association of Oregon…includes link to OregonWorks for Jobs
10 Ways to tweak your tech resume, also on lifehack.org, but referencing eWeek. However, I disagree with the “lose the one page only rule” tip. When reading resumes I literally throw out all but the first page. If the candidate doesn’t engage me by the bottom of page one, it wasn’t meant to be. I’m not completely clear how to reconcile that with the keyword problem, though.

There’s a ton of job web sites out there (hotjobs, monster, dice, etc.), but a quick google search will find those pretty quickly.

Intel update, part deux

Wed, 04 Oct 2006 14:57:00 -0700

Well, I remain gainfully employed, although I will be doing something different. What exactly, I’m not yet sure. What I do know is that every manager from my formerly 3rd level manager down has been affected. One layoff, 2 redeployments, one transition to non-manager employee, and one move to another organization.

So far, there are 28 people that I know that have been “redeployed”. It doesn’t feel like I know 300 people at Intel to really make that a 10% cut, but any way you cut it this is a difficult time. It’s Wednesday of employee announcement week, but hopefully most everyone now knows their fate and we can move on. Next week is re-organize week, so better (or at least different) news is still to come.

Seriously?

Tue, 03 Oct 2006 17:47:00 -0700

I just went to find the phone number for a local Indian resteraunt called the Curry Leaf. I found the phone number on their web page, but also noticed that they have a blog. Not sure what a resteraunt needs with one…I can see a chef, maybe, but just the resteraunt?

Also enjoyed their contact us page, which tells us what their website URL is. Of course if I’m on their web site, wouldn’t I already know that?

Hmm…anyway, they have good Indian food. ;-)

Odd marketing move

Wed, 27 Sep 2006 16:42:00 -0700

I came home today to find a UPS 2nd day air envelope at my front door. Not expecting anything, I went ahead and opened it up to find a letter from Verizon trying to get me to sign up for their FIOS service. Well, I already have FIOS, it works pretty well now that I think I have the kinks out of it, and so I’m not sure it was worth the crazy amount it probably cost to do a direct mail solicitation using UPS 2nd day air.

Of course, I’m blogging about it, so maybe it wasn’t a bad move?

21 hours and counting

Wed, 27 Sep 2006 15:04:00 -0700

That’s how long it’s been since:

A router reboot
An IP address change on Verizon

I found out the other day that the reason my IP address kept changing (and subsequently knocking me off the phone), was that I didn’t have Verizon’s VDI-624 router as the first router in my setup. The VDI-624 is the same as a DI-624 router from DLink but has specialized firmware for Verizon. The interface looks the same, but they must truly have done something different to it.

Anyway, after fixing the IP address change problem, my old problem of router reboots came back. I run both eMule and Bittorrent, and so I have an insane number of connections open. For some odd reason, the router could handle the connections when my Vonage router was closest to the Internet (even though it was in the Vonage router’s DMZ), but not when the VDI-624 is closest to the Internet. Odd. I didn’t really need Bittorrent running so I shut it down completely and all is well, although it would be nice to have everything running, so maybe I’ll play with it another day. Not today. Today, everything is working as designed. That’s a good thing.

Intel update

Mon, 25 Sep 2006 09:07:00 -0700

I usually try to refrain from posting work-related stuff since you quickly start riding a fine line for what content is acceptable, but today I’ll go ahead. We are (still) in the middle of our “restructuring” (a.k.a. 10% downsizing). Quick update from my seat of visibility:

April - first “efficiency” announcement internally and to wall street. Announced to complete in 90 days (August).
June/July timeframe: Number of “redeployments” affected several groups. A few friends of mine were affected by this. Since this was the first action, everyone I know was able to find another job inside Intel.
July: 1000 Manager reduction. My manager and my 3rd level manager were both laid off.
August: Announcement that analysis was complete on time, but execution on that analysis would take until the end of the year. Redeployments would occur in September.
September: Redeployments will occur WW39 for managers and WW40 for other employees. In normal terms, WW39 is the week beginning September 25th and WW40 is the week beginning October 2nd.
Last week: My admin was “redeployed” and I’m told that our group will have a 20% reduction. Another group I work closely with will have a 75% reduction.

I’ll keep a lot of my analysis of this to myself for the time being, but will mention that a redeployment at Intel means that you are given several months where your job at Intel is literally to find another job at Intel using our internal system. Alternatively, you can opt to leave immediately and take a severance payment. There are some internal positions open, but not many, so a redeployment effectively means a lay off right now.

For those reading that think that the process is very long, I agree, but will offer that it appears as though management is trying to work the logistics of the process in such a way as to make sure that people that are affected are eligible for a few bonuses. If true, that’s a nice silver lining to this particular cloud.

That's a bold move...wonder how that'll work out?

Mon, 11 Sep 2006 12:27:00 -0700

So I’ve just finished making the switch for my default browser on my work system. I wanted something other than Internet Explorer primarily for tabbed browsing purposes, but secondarily to help reduce the amount of memory that IE takes up.

My first switch was to Maxthon, which internally runs IE but handles tabbed browsing and mouse gestures. I now pretty much expect my browser to handle mouse gestures and get frustrated if holding the right mouse button and moving down and then right doesn’t close the tab.

Maxthon was really very good, except that there was no plugin available to handle Google Bookmarks, which solved another thorny issue for me.

Firefox sounded like a great solution except for the number of sites I use (primarily inside Intel) that require ActiveX controls and other IE specifics. I bumped across the solution to this recently when running into an extension called IE Tab. This allows you to load IE within Firefox, mostly taking care of those issues. It hasn’t worked with Test Director (and neither does their plugin from 1999), so I’m not entirely off IE, but I’m nearly there.

Oh, and NTLM authentication for the tons of Intel sites that need it? Yeah, it’s got that…

On the topic of flash

Tue, 15 Aug 2006 15:07:00 -0700

So smaller files are better off on flash memory? Small files:

Source code
Most web stuff (HTML, CSS, JS, etc., etc.)
Lots of miscellaneous docs (word, powerpoint, exel)

Sounds to me like the best way to work on these things is to use flash memory as your primary work area and copy the files to hard disk only as backup. At least until Hybrid hard drives solve the problem…

Back in a flash

Tue, 15 Aug 2006 12:02:00 -0700

There’s a new performance feature in Windows Vista that has piqued my curiosity. The feature is based on pre-fetching (where programs are loaded into memory or in an area on your disk before you run it, because the system knows you usually run the program).

There are some other enhancements to the pre-fetch feature, but the interesting one to me has been coined as ReadyBoost. This feature uses flash memory (like a USB drive or compact flash card), and loads the pre-fetch information on the memory. The concept is that this will make your system faster.

Knowing that USB drives are slower than hard drives, I found the claim curious. It turns out that the way they access the data must be in smaller chunks. While flash memory is slower, the access time is much faster. There are several assumptions that vary the analysis fairly significantly, but my most pessimistic assumptions say that any file smaller than 100k is better off on flash memory. My most optimistic assumptions say that any file smaller than 500k is better off. Over those sizes, a hard drive will be faster (assuming that you keep your disk defragmented).

Your data...available and protected

Fri, 11 Aug 2006 17:21:00 -0700

So I made the plunge and purchased a NAS. I got a Buffalo Technology product, but not the one that I previously mentioned. The difference between the Linkstation Home Server and the Gigabit Linkstation is DLNA certification for media streaming to a very few certified players, which seems to add about $75 to the cost. In 5 years I may regret it, but right now, I’m pretty happy with the HD-HG300LAN. I’m also somewhat future-proofed with the Gigabit option.

I moved all the data from my external USB drive and scheduled backups onto the USB drive (my external USB is now strictly a backup device). So far, so good. The downside is that the data transfer had to occur over the network since the NAS web interface doesn’t allow any file manipulation. The firmware (v1.42) is pretty rough around the edges, and setting the security up hasn’t worked so well, but everything else has worked as advertised, and pictures of Kathryn are now backed up.

It's been bugging me

Thu, 10 Aug 2006 14:04:00 -0700

I had seen a website that contained a list of login credentials for many of the “free but registration required” web sites. After more googling than I really would care for, I found it. www.bugmenot.com

dotnetnuke on GoDaddy

Tue, 08 Aug 2006 12:25:00 -0700

In my last post I mentioned bethanycommunity.net web site. In getting it working, I had to get dotnetnuke installed and running on the root directory of the site. Doing this on godaddy was quite an interesting feat and required some changes to the dotnetnuke source code. I hope their latest version helps fix that problem. There are some instructions on doing this here. I found they got me close, but the installer was still a problem, and hence the (temporary) source code changes to skip over some of DNN’s more dangerous install behavior.

Some people think that this is GoDaddy’s problem and people should be able to create very insecure sites, but I side with GoDaddy that normally applications should not need to update their core configuration files and place files in their production directories.

Local issues

Tue, 08 Aug 2006 12:17:00 -0700

I’ve volunteered some time to get a website up and running for a local organization. The Bethany Coalition was renamed to the Bethany Community Network, and was originally formed to help out with planning to accomodate more people moving into the area. Portland has an urban growth boundary that helps to confine population and prevent urban sprawl. In our area, the boundary was expanded to enable more people, but that means new infrastructure needs to be in place. You can read more about it on their (my) web site. Feel free to comment or email me directly with suggestions for improvement.

Windows annoyances

Tue, 08 Aug 2006 09:05:00 -0700

Why is it that after roughly 18 years of work on the Windows NT family of operating systems, Windows XP still forgets my desktop background picture?

As I had deleted the original file, I had to go spelunking today to figure out that it’s copied to C:\Documents and Settings\[username]\Local Settings\Application Data\Microsoft\Wallpaper 1.bmp. The most frustrating part is that I could see the picture, but only when logging out/shutting down.

Ruby on Rails?

Wed, 02 Aug 2006 08:18:00 -0700

I got a chance to check out Ruby on Rails the other day. That basically explains it. Rails is a set of libraries that use the programming language Ruby to make web programming very, very productive. I’ve heard of defections to RoR from .NET and Java. I’m sure there are some, but the stories appear to be anecdotal. Anyway, it looks promising, and I hope the competition helps improve the productivity of all languages…

ObjectDataSource limitations

Wed, 02 Aug 2006 08:12:00 -0700

One of the more exciting things in Visual Studio 2005 for me was the idea of an object data source for grids. Well, I’m not doing a lot of hands on programming right now, and I finally got around to using it. I’m not that thrilled. Here’s a snippet from a help file that is my current source of frustration:

The ObjectDataSource control will create an instance of the source object, call the specified method, and dispose of the object instance all within the scope of a single request, if your object has instance methods instead of static methods (Shared in Visual Basic). Therefore, your object must be stateless. That is, your object should acquire and release all required resources within the span of a single request.

This pretty much means that they expect an object to be a pass through to some other data source (XML, DB, etc.). I’m just capturing some session information right now. I find it easier to create/populate an object and databind directly then try to make an object that’s specific to this problem. I think for the time being I’ll go that route…

Efficient Markets?

Wed, 02 Aug 2006 07:58:00 -0700

I don’t think so. The theory/hypothesis is full of crap. We learned this in our finance classes in our MBA, but I don’t buy it. There are several versions of it, but even the most weak of the arguments claims that technical analysis won’t work. From what I’ve studied of technical analysis, it’s trying to use numbers and history to predict aggregate investor psychology. A simple form would be that if a stock is at it’s 52 week high, more investors think it’s time to take profits than investors think it’s time to get into the stock, so the price will probably go down.

Efficient theory claims the market is a machine that absorbs information and adjusts the prices within seconds to account for that information. Nothing else matters. I think the market is made up of people, and people have feelings, and stocks become either overhyped or forgotten. And when that stock hits it’s 52 week high, I know I start to consider taking my money off the table…

I like Warren Buffett’s feeling on the matter (which comes from Benjamin Graham). Take a look at this commentary also.

Mad money?

Wed, 02 Aug 2006 07:50:00 -0700

I’ve found Mad Money to be an interesting and entertaining show. I am watching more regularly now that my daughter has arrived and I need something to do while she sleeps in my arms, and with the summer here, this is not bad. Anyway, I don’t always agree with Cramer (the host), but where I have agreed and taken action on his recommendations, it’s worked out well. The latest trade is Valero Energy (VLO), bought before Q2 earnings announcement yesterday. Earnings came out as (Cramer) expected. Yesterday wasn’t a good day for the market, but today both the market and Vallero seems to be heading in the right direction…

I recommend the show, but only if combined with your own research and analysis. Make sure you’re not “drinking the Kool-aid”.

Sometimes, it's not what you know...

Mon, 31 Jul 2006 09:55:00 -0700

Jeff told me he was going to blog about this, but it doesn’t look like he’s done it yet, so I will. Jeff has been looking for a notebook recently, and since I work for Intel I’m pretty close to that particular market. I’m also working on the Verified By Intel® program, which deals directly with the notebook market, so I’m pretty clued in.

So Jeff emails me on July 3rd to ask me what the T2050 processor is. He found a notebook that includes the processor, and can’t find any information on Intel.com…I verify his results, find processor information on wikipedia, and then the fun begins. I can’t find information anywhere internally, which is a huge problem for me because if I can’t find information, that means that Intel can’t process returns on the product and there is no support documentation on the product.

Jeff emails customer support, and after being routed around (support is looking at the same things I am, so they’re running into the same problem), basically gets back a non-response around July 11. I have a few internal contacts that Jeff doesn’t have access to, so I email them, but Intel is in the middle of a major launch at this time, so I don’t get back any response. Finally I hear back on July 25th, and the information I get back is somewhat disturbing. I find that:

The processor is “off roadmap”, meaning that it wasn’t planned at all, but rather put onto the market at the last minute, without any supporting documentation.
It’s OEM-only (I support “boxed products” - these are products that you can buy in a box online or through a retailer). Being OEM only, you can only get the processor if it’s already in a computer. This is why it’s not in my tool for returns, but it still should be in for technical support…
The person responsible for worldwide support of the processor didn’t know anything about the processor until he bumped into one at Circuit City.
This apparently happens all the time with desktop processors.

What a way for a company to support its products. Hopefully the drive to efficiency will help…

Long blog posts

Wed, 26 Jul 2006 15:53:00 -0700

I generally dislike really long blog posts. I think my last two were too long. So this one won’t be.

Software Update annoyances

Wed, 26 Jul 2006 15:37:00 -0700

My last post also reminded me that I’ve been meaning to post on another subject - that of software updates. These are (sometimes) necessary and usually desirable functionality, as long as they don’t get in the way of the user trying to do their job. As mentioned previously, I think Adobe does a great job showing how not to do this (modal dialog box, even when viewing in a browser, usually asks to install a bunch of unneeded crap, etc.).

One thing I see consistently done wrong in these update programs is the fact that they tend to ask you to update when you start the program. If a user started a program, do you think they want to update the program, or use the program? (that’s rhetorical) Unless the update is critical, the update process should ask to update the program and/or its components when the user requests shutdown of the program. Here’s the complete guidelines that I think should be followed:

If the update isn’t critical, wait until shutdown to request the update. “Critical” updates include:

Security-related issues that can expose the user to remote attack
Stability issues that the user has experienced. You would know this only if a crash reporting facility exists and the user has opted into the use of the facility.

If critical, state so and offer the update on application start

On shutdown, ask the user to update, but close the application if no reponse has been received after some time (how long?)

Only include updates for existing components - don’t ask the user to install Google Toolbar (unless of course the application that they’re updating is Google Toolbar)

Allow the user to choose which components to update. It’s ok to use an opt-out model for this. Include a “don’t ask me again” facility along with a way for the user to go back and install the “don’t ask me again” updates (e.g. through a specific update application menu item)

Try to avoid forcing an application restart. If you’re shutting down the application it may not be a big deal, but if the user selects that menu item or the update is critical, this becomes important. It’s really important if the application is a service or an operating system. Yes, Microsoft does a horrible job at this…

If you have multiple applications in your company, use a consistent service that can update all applications. I do give credit to Microsoft on this one as they are working on getting a single update service. Ideally, there would be a single service/paradigm used across companies (e.g. MSI/Installshield for installations). Market opportunity?

Of course, I could be wrong, but I don’t think I am. Let me know.

Programming that screws things up

Wed, 26 Jul 2006 14:22:00 -0700

As I am waiting to respond to a friend of mine today because my browser won’t respond, I’m reminded of a few really stupid things that hopefully will become less common in software in the future.

The specific incident today was that I wanted to click on a link (Outback Steakhouse Menu), but the browser stopped responding. Was my computer slow? No. Was it doing something else to the point where it couldn’t handle my command? No. What happened was that I had been downloading a 50MB file from a web site. The other browser window just finished downloading the file…so the last step in the process is to copy the file from the temporary directory to the final download directory. During that step, apparently all other browser windows stop operating.

These issues make computing frustrating, and to the casual user, makes the computer “slow”.

Other annoyances:

Modal dialog boxes. These are barely tolerable under normal circumstances, but are horrible in applications with multiple windows when the modal dialog box requires input before any operation can commence in any window. Adobe Acrobat does a fantastic job with this one…they have a window come up for their product updates that sometimes pops under the browser when you view a PDF through the browser. It can hang Firefox and can appear to hang Internet Explorer.
Waiting to have it “perfect” before displaying results. Windows explorer is a good example - some directories will take 30 seconds or more because I have large files and it wants to do something (to prepare thumbnails maybe?). I don’t even need the icons right away - just show me the file names and fill the rest in on your own time!
Lack of user feedback. This just happened to me also. Click on a link (Quicktime movie) and nothing appears to happen. It’s actually downloading but I don’t have any feedback, so I click it again, and the download restarts, but I am completely unaware of this. I see others struggle with this one. My favorite is when someone launches a really big application but there’s no feedback, so they launch it again, and repeat until eventually the computer brings up the application, but then brings up like 10 more windows and is completely choking on memory.

Other examples aren’t coming to me, but I’ll try to post when I run across one. Feel free to leave comments with other good annoyances that appear to slow/stall the computer, but really aren’t.

IPOs - not hard to call?

Fri, 14 Jul 2006 10:07:00 -0700

Maybe this is just a recent phenominon, but I’ve been watching and/or participating in IPO investing (sometimes trading) for the last year or so, and have a 100% success rate in making decent money. We’ve been selective, but made investments in Under Armour (UARM), Master Card (MA), and thought really hard about Sealy (ZZ) and Burger King (BKC). Here’s the formula I’ve (we’ve) been using:

Absolutely must be a well-known company. If the company name isn’t recognized, then I don’t usually give more than a cursory glance at the rest.
Business model needs to be solid. We passed on Vonage (VG) because they have little to offer customers to stay with them, and the big incumbants can (and are starting to) roll out similar services as comparable price points. There’s no reason for customers to stay.
The wife needs to agree. Not to be flippant here…a second or third person brings a different perspective and will often kill some reasonable concepts (in our case Sealy and Burger King).

UARM and MA were both fairly short term investments, and you can certainly argue with me there. UARM was held for about a quarter until I got sufficiently concerned about Nike coming in to destroy them.

MA is a decent long-term investment, but I had to go on margin to buy the stock, so I sold after a month and having made some significant money. Even without the margin issue, our #1 rule has been “don’t get greedy”.

When technology and consumers work well

Thu, 06 Jul 2006 18:32:00 -0700

Now that I have a fantastic reason to use the new camcorder, I have been making some recordings and otherwise playing with the features. I recently posted two videos to Youtube.com, and the front-to-back process was really straight-forward…technology should continue to put consumers at the center of their world like this:

Plug camcorder into computer, turn it on
Load Windows Movie Maker
Tell WMM to import video from the camcorder
WMM detects the clips on the camcorder
Take each clip you want to save, save the file as a seperate WMV
Upload WMV file to YouTube

While the above process is really cool, here’s how I could see it optimized even further:

Plug camcorder into computer, turn it on
Load Windows Movie Maker
Tell WMM to import video from the camcorder
WMM detects clips, but also stops when hitting the end of the tape that’s been recorded on
Take each clip you want to post, post the file (menu item/context menu) directly to YouTube - if login required, WMM can request, or integrate with the upcoming Infocard

When technology and consumers collide

Thu, 06 Jul 2006 18:29:00 -0700

So I ran into a really weird situation with the camcorder I recently purchased. The camcorder has two data ports, one is Firewire, and one is USB. The camera itself can do three data-related things:

Transfer recorded video to the computer
Transfer pictures to the computer
Use camcorder as a web cam

In order to do #1, you need to use the Firewire port. This seems logical but a bit antiquated. Originally, Firewire could handle speeds of 400Mbps where USB could only handle 12Mbps. Apparently Sony decided it was easier to force folks to use the Firewire port than to have to explain the differences between USB 1.x and USB 2.0, which can handle speeds of 480Mbps.

All fine so far. I bought the firewire card and cable and installed it into my computer. Now here’s where things get weird. Firewire does not allow you to do #2 and #3 above. So now I need to actually use both the firewire and the USB cables in the computer if I want to do all three operations. How weird. Maybe Sony is trying to transition off Firewire in favor of USB?

Whatever the reason, it’s a very awkward customer experience, and I personally think they should stick with USB only and suck up having to explain USB 1.x vs. USB 2.0. I would think they’d enhance the customer experience while reducing both manufacturing and support costs.

Bookmark nirvana

Fri, 23 Jun 2006 13:21:00 -0700

I’m finally getting bookmarks under control, and I can’t believe how much better life is. When you use Internet Explorer for stuff that doesn’t like Firefox (and for most things work-related), and Firefox for normal browsing, and you have multiple computers, things get pretty unweildy. Now, thanks to Google, my problem is mostly solved.

There are several bookmark managers out there, but only Google Bookmarks seems to have solved my main problem and has support for both Internet Explorer and Firefox. Here are the caveats:

Internet explorer support is through a beta version of the Google toolbar
The Google toolbar for Firefox is still back a few revisions, so I need a third party firefox extension to support it.
Maxthon, the software I use to get Internet Explorer in tabs until IE 7 comes out, doesn’t support the google toolbar, and there’s no good addon for Maxthon directly. I’m starting to use IE (without Maxthon) a fair amount now, and it’s the one big downside to this bookmark thing.

Questions and Answers are like footnotes in annual reports

Tue, 13 Jun 2006 08:58:00 -0700

I was watching a webcast the other day (topic irrelevant) and it occurred to me that what I really like about webcasts and earnings conference calls is the Q&A section at the end. All the pre-canned stuff in the beginning can be so filled with political spin it’s hard to get a handle on what’s really going on. Q&A (either participating in or listening to) provides the opportunity to cut through some of that and get the real information.

Outlook 12 goodness

Tue, 13 Jun 2006 08:54:00 -0700

I haven’t actually used Outlook 12, but I just read a short list of Omar Shahine’s favorite features. I look forward to the new features (there is also a document auto-preview that looks cool but I really look forward to figuring out how it’s breaking my system).

I find the color categories feature strangely familiar from gmail…maybe they’ll incorporate that feature into Windows Live and really copy Gmail. ;-)

In MS’s defense, at least they’re not too proud to copy features (or buy companies) when they’re out-innovated.

Camcorder purchase

Mon, 05 Jun 2006 13:45:00 -0700

I just purchased a Sony HC36 and documented the experience for a couple friends, so I thought I’d share it here. Here’s the lowdown on the process (skip to step 9 if you want to skip how I figured out what camcorder to buy).

Consumerreports.org led me to a JVC camcorder, and any one of several Sony camcorders. Sony is the best overall brand, while a couple individual JVC, Panasonic, and Cannon camcorders rate well.

I looked at Amazon to confirm the JVC camcorder, and was surprised to find lots of poor reviews for it. I eliminated it as a candidate.

Took a look at CNET.com for the other camcorders. They had some reasonable advice, but being a tech-focused review site, they were pretty much only interested in “the best of the best of the best – sir!”. Their reviews basically pointed to buying the most expensive camcorder (like the Sony High-Def camcorder for $3600). I was able to at least discern some of the differences between the Sony units from CNET.

Narrowed search down to Sony basically due to a) price (this eliminated the Panasonic) and b) a few features (like webcam abilities that eliminated the cannon).

Googled for the differences between models and determined the following:

Of these 6 models, I determined that there is very little difference in the 2x/3x series. The 42/46 is significantly ($100) more expensive and has better technical specs, but the 2x/3x differs in just a few features and the core video capture capability is the same.

The 32 was cheapest on Amazon and I tried to do better on eBay, but was unsuccessful.

The deal I saw on Amazon went away and a friend pointed me to Pricescan.com. The price graph on the 32 showed a spike in prices in the current week, confirming prices were up overall on the 32.

Based on pricescan, the prices between the 32 and 36 were pretty much even, so I looked around various sites (Amazon, buy.com, overstock.com, outpost.com, shopping.yahoo.com) to try to beat pricescan. I was able to beat pricescan.com, but only by $2.50.

I used shopping.yahoo.com to check out merchant reviews and make a selection. The second best price came from the lowest price vendor on pricescan.com. They had a good review and Yahoo buyer protection and I was going to go with them until their website had a technical difficulty. I went to the cheapest vendor (4.5 out of 5 stars, but no Yahoo buyer protection) and placed the order there.

I needed a set of accessories, and already knew where to get the best deals, so I went straight to newegg.com. I already had a DVD burner and media, but I ordered (total cost ~$75):

Firewire card for the computer
Firewire cable (6 pin to 4 pin)
25 pack DVD movie cases
512MB memory stick pro duo (I hate how Sony has proprietary memory formats).
6 pack of mini-DV tape

I put together a saved wish list at newegg.com that includes what I ordered as well as the internal DVD burner and media I would order today if I needed them. If you need the external version of the burner it’s probably another $10-$15. Also note that the memory is optional – as far as I know, the camera only uses the memory stick for still photos. I’m also not 100% sure I don’t need a USB cable to do the webcam functionality, but right now I’m assuming I can use the firewire cable for both. You cannot use the USB cable to transfer video with that camera, you’ll need the firewire.

I passed on ordering a camcorder case. My plan there is to get all the stuff in house and then go to target and see what would hold all of it without taking up a ton of space. I did not see a significant savings by ordering the case on the Internet. I also passed on tripods, lens filters, etc. I just don’t think I’ll be going that professional, and if I did, I’d spend a lot more on the camera (the $800 price point will get you to a semi-pro level). I did get a phone call from the people I bought the camcorder from trying to sell me all the accessories. I told them “I had an older handycam and already own all the accessories”. That got them off my back pretty quickly, but I did spend the extra $30 for a 2-year warranty. I have heard rumors that the touch screen tends to wear out and it is not covered by Sony’s warranty.

The way it should be

Fri, 02 Jun 2006 09:42:00 -0700

Blogging from a coffee shop in downtown Portland for free

Is it really that much to ask for?

Tue, 23 May 2006 12:15:00 -0700

So I’ve been keeping my eyes open for a cheap but somewhat flexible NAS solution. Quick rundown of my requirements:

Stores data
Allows always ready backup (usually this translates to RAID 1 mirroring)
Allows connection via SMB, so I don’t have to install software from a hardware manufacturer (this is my best known method for not getting weird errors)
Plugs into wired Ethernet
Expandable
Free (or at least cheap)

The other day I ran across what appears to be such a device, from a company called SimpleTech. Now, what’s amazing to me is that there is very little competition for this. I did manage to find something very close from Buffalo Technology, and it may even be better as it does support streaming media. However, it’s backup strategy is a scheduled backup rather than RAID 1 (which might be just fine).

Other products tend to be a couple hundred dollars more expensive and include a bunch of stuff that is nice, but not necessary to a consumer, or in my opinion, to a small business (<10 employees, probably retail, not a lot of discretionary funds). I’m going to give it a little time and then go get the Buffalo or SimpleTech.

Gas prices

Mon, 22 May 2006 10:24:00 -0700

One of my soapbox items for the press is their failure to report historical prices/costs on a real basis. This is a disservice and misleads people. Over the last year there’s been a lot of discussion about oil prices. Today I searched/found a good/short article on gasoline and oil prices from the American Petrolium Institute that reports on a real basis. The bad news? On a real basis, we are indeed near (but not quite at) historical record highs for gas at the pump…

The goodness of lots of bandwidth

Mon, 22 May 2006 10:15:00 -0700

So FIOS appears to be fairly stable now with the exception of random router reboots about once/wk. I won’t rant on that right now, but I would like to quickly brag. Last Friday, Kelly and I were working from home, and for an hour, we:

Both used the phone (her on Vonage, me on Skype using SkypeOut, which is free until the end of the year to US/Canada phone numbers)
She used VPN and Netmeeting
I used a form of VPN and ran reports/data extracts during the meeting

The service worked fine the whole time (with the exception of a random router reboot…grr).

Verizon FIOS Update

Tue, 09 May 2006 16:24:00 -0700

Ugh…so some mixed results in the really high speed world. Got FIOS installed cinco de Mayo and all was great - for a day and a half. On Sunday we started having problems (no access at all), and to make things worse, the phone number that was given on literally every page of our welcome guide and used in the “Need Help? We’re on call for you 24x7” section of our welcome letter got us nowhere. We were both completely unsuccessful finding the right path to get to 24x7 support, and a complaint letter has been drafted.

On Monday (afternoon) I got a chance to call during business hours and got routed around. I now have the super-secret tech support number of 888-339-7333. I’m not sure how this relates, but there is another number of 888-991-4999 that someone called “FIOS Service” (compare with “FIOS Tech Support” for the other number). They fixed us at the time by doing something on their end, which I now suspect was that they rebooted the box on the outside of the house. I asked at the time, but the guy on the phone didn’t know - he was relaying between me and the network people.

So, all was good…until this morning at 6AM when I had a meeting with India. Using Vonage, not having broadband is like, really a downer, and I went back to my Comcast broadband for the phone call (and for data while working this morning). This time, I had download speeds of anywhere between 2-5Mbps, but almost no upload (about 60kbps). The speeds kept varying wildly, and over the course of the next hour or so, they gradually got a little better. Latency was no problem at all, but I was getting a lot of dropped packets as well. This was really weird.

I called Verizon about 8AM and they rebooted the box outside, and it was better, but still not “right”. It took about 40 minutes for me to convince them that it was not a problem with my computer (they remote connected to two of my machines and ran optimizers and stuff). In their defense, everything they could see (which was to the box outside the house) looked ok on their diagnostics. They finally decided it was weird enough to send someone out. Oddly, as the morning drug on, my speeds were still intermittent, but they were slowly getting better.

The technicians (yeah, two of them - pretty cool) got here about 10:30 and they took a look at the box outside as well as one of my machines. The one technician ran a ping test on the first hop router while the guy at the box outside my house noticed that the wires in the RJ45 connector didn’t have a firm connection. Bingo - this explains the problems! In cold weather (overnight, early in the morning) the metal wires contracted, making the slighly unfirm connection really, really shaky, to the point where it just wouldn’t do anything anymore. Upload speed was effected because it was one of the TX wires…pretty wild.

So they fixed the connector problem, but the speed is still not quite “right”. They take a closer look at my setup, and they say, “who installed this? The installer used internal wiring, which is Cat 3, not Cat 5!”. After wiring up a new (Cat 5) outlet the problem is now fixed and I have some killer broadband.

As a reward for any west coast people reading this far, the Verizon guys on the way out the door mentioned to me that on the east coast, what I pay ($44.95/mo) gets you 20Mbps/5Mbps (vs. my 15Mbps/2Mbps), and that should be rolled out here as well at some point….

Connected, or not?

Sat, 29 Apr 2006 14:31:00 -0700

I constantly struggle with the concept of connected vs. unconnected operation. One thing I hate when traveling is that I don’t have access to many common applications (the system I am responsible for at work, our bug tracking system, many, many random web sites, etc.). However, I also do not like the idea of installing all kinds of applications on my notebook and syncronizing all the time.

For applications, I’m starting to come down on the side of building interfaces based on the client. If your clients are all Windows, it probably makes sense to go with a .NET ClickOnce application with SQL Everywhere as a local data store (of course, you get to have fun building in all the syncronization). If your clients are not all Windows-based, then go with a web application, and if the costs justify the it, build the click once app as well for convenience.

Similarly (and possibly a more complex problem), what to do about email? Do you send that large attachment out? If you do, your coworkers/friends/whatever will not be able to edit the document and easily keep track of changes - you end up merging by hand constantly. If you don’t, your recipients do not have offline access and may have a difficult time finding the document again. I don’t have a good answer for that, but I suspect that a really good answer would be worth money to someone…

Got FIOS?

Sat, 29 Apr 2006 14:18:00 -0700

Well, the trucks have rolled through the neighborhood laying down fiber optic cable through the streets. While I don’t have FIOS yet, it will be installed May 5th. For the uninitiated, the service is high speed brodband Internet access provided by Fiber Optic cable directly to the house. Instead of a cable or DSL modem, what gets installed in your house is an ethernet jack. I like the service for the following reasons:

It’s fast (15Mb/sec, which is triple the speed of cable service here)
It’s fast (2Mb/sec upload speed, which is 6 times the speed of cable service here). Why do I care about upload speed? When working from home, talking on Vonage (over the Internet) and sharing applications in Netmeeting becomes painful.
It’s priced well. For $1 less than my already-discounted broadband cable service, I get the 15Mb/2Mb package. If I were even cheaper than I already am, I could have 5Mb/2Mb service for $10 less. That would give me roughly the same service I have now, but would vastly improve my upload speed.

I may have splurged a bit on the 15Mb package…not sure I’ll use all the download speed. But guaranteed, it will be nice.

As an aside, the power of marketing and brand recognition is really evident in this space. I could pay 3x as much with Comcast (or Verizon) for the same service I get from Vonage. Verizon FIOS is very new, so they are priced so well that there’s just no reason to switch. Either you get better service for the same price (like me), or you save $10/month. With 1 month free and a 30 day money back guarantee, there is no customer switching costs. It will be interesting to see how Comcast reacts to the competitive threat when FIOS starts getting mindshare…

Gas prices and irresponsible reporting

Fri, 28 Apr 2006 08:12:00 -0700

There’s been much ado about gas prices lately. I’ve seen a lot of irresponsible (or just plain ignorant) reporting on the matter. Headlines like “Chevron Earnings Soar 49 Percent to $4B” and “Exxon Mobil 1Q Profit Up 7 Pct. to $8.4B” provides very little context, and if you actually read the report, you’ll notice a lot of dollar figures. I have not seen (or heard on the news) a single non-financial report of gas industry profit that even mentions margin. I might be picking on AP a little here, because both my examples were AP, but they are one of the most read news sources, so they have a greater responsibility to get things right. Anyway, the whole thing reminds me of deficit headlines that don’t compare deficit as a percentage of GDP.

This post is about bad reporting, but I’ll mention that my opinion on the gas thing is that gas is a commodity and that as such, the oil companies are price takers, not price makers, and so they have very little control over these things. An 11% (Chevron)/16% (ExxonMobil) operating margin is pretty-freaking far from “windfall profits”, and appears in line with other commodity industries like steel.

Smartphone Applications

Sun, 23 Apr 2006 11:52:00 -0700

Hmm…so Smartphone 2003 (a.k.a. Windows Mobile 2003 a.k.a. Windows CE 4.21) doesn’t come with a version of notepad for short notes.

Problem now solved. I wrote a quick notepad application. Curiously, Smartphone 2003 doesn’t have a standard control for navigating the file system. The guys at OpenNetCF have some great extensions, and although I can’t find it right now, there is are File Dialogs to help you open and save documents.

I also decided to write a stopwatch application. The application has standard start/stop/reset/lap functions, but also adds logging to a file (no, you can’t change it). It also has a unique feature that helps explain why I built it: It will time contractions (and display the last hour’s average frequency and duration), show you the last 5 frequency and durations of the contractions, and tell you when it’s time to go to the hospital. I expect it to be quite handy around the end of June…

Yahoo! Mail Issue Resolved

Sun, 09 Apr 2006 13:15:00 -0700

As mentioned in a previous post, I had some significant issues getting Yahoo! Mail up and running. The issue is now resolved. For the record, I believe that the issue stemmed from the fact that I established my Yahoo! ID back in 1995 (or thereabouts) and did not sign up for mail service on the ID at the time. I did not really want to establish a new ID for the test. Anyway, here’s the scoop:

February 25th: Original ticket submitted
February 27th: Stock answer that the problem is temporary, and maybe I should click refresh on my browser. Not sure why it took two days for what an autoresponder could have done, but whatever. Responded that I tried IE and Firefox, logged out, logged in, etc., etc.
February 28th: Received response that they are looking into the issue.
March 5th: Responded to Yahoo! asking for an update. Received response that they’re still working on it.
March 22nd: Responded to Yahoo! asking for an update.
March 29th: After a full week, I received a message saying “the issue may have been transient and appears to be resolved now”. My conclusion at this point - Yahoo! did not actually look into the issue over the entire month. I’m basically back at “please hit refresh”.
April 1st: Hadn’t checked into my email, so I read the March 29th message and responded with “This is not yet resolved.”
April 2nd: Finally I seem to get someone’s attention. They ask me for a bunch of account details (not sure why they need this, but they’re finally looking at my account setup rather than assuming I’m reporting a general server outage).
April 3rd: Respond with account details
April 3rd: Yahoo! replies to my response asking for account details. After a month of back and forth blaming me for not hitting refresh, now they prove they can’t even read.
April 4th: I send my reply “I already did. Please see original message below”
April 6th: Michelle replies to me stating a definitive, “We have completed our investigation of the issue you reported, and were able to identify the cause and take corrective mesaures to address it…”. This sounds positive!
April 7th: I verify that I now have a Yahoo! mail account and can log in. I send a “It’s resolved - thank you!”.

The entire time I had hoped to actually call them so I could convince someone on the phone that this is an account issue, not a server issue, but alas, no support phone numbers were to be found. I actually considered walking to their offices near Kelly’s work either to explain the situation, strangle someone, or both.

Anyway, in a record 7 weeks, Yahoo! was able to fix my problem. One good thing in all this is that Yahoo! sent me a survey after nearly every response. I sent some very strong messages in both the scores and the write in comments. I hope they read them.

April Fools Day comes early@Intel

Thu, 30 Mar 2006 14:39:00 -0800

We had a fantastically well done April Fools joke played on the 100k or so Intel employees today.

Everyone in the company received an email stating that there was a major new deal between Apple and Intel, and that executives would host a webcast to announce it today at 8:30AM Pacific. The webcast has been replaying every half hour since then. Very little details have been posted about the content of the announcement, and the article on our internal web site states that the announcement will be made public to the stock market tomorrow (Friday March 31st).

I missed the original webcast (conflicting meeting), but caught it later in the day. Keep in mind this is an April Fools joke…but the executives announced a merger between Intel and Apple. They demonstrated the new name (mactel), the new logo, new tag line (leap different), and talked about transition plans, and why they decided to merge. By the time they got to uniforms (black turtleneck) and Intel, wait, sorry - mactel values of “Be Cool”, it was pretty obvious it was a joke.

Intel does a great job with April fools every year. Normally they turn our internal website into a huge pile of fake news, interviews, etc. This year, April 1st lands on a Saturday, so they thought they’d get us all a little early. Since it wasn’t the standard fake news that we’ve now become accustomed to, it was extremely effective. We may be a big company, but we do have fun every once in a while.

I still hope they do the fake news…it may even be worth logging into work on Saturday just to read it. ;-)

Disclaimer: The content of this message is my personal opinion only and although I am an employee of Intel, the statements I make here in no way represent Intel’s position on the issue, nor am I authorized to speak on behalf of Intel on this matter.

Live Custom Domains vs. GMail Hosted

Thu, 30 Mar 2006 10:36:00 -0800

OK, so this is a bit bleeding edge, but very, very cool. Please take this mini-review with a very large grain of salt because both services are evolving fast.

Both Windows Live Custom Domains and Gmail For Your Domain provide the same basic functionality, which is the ability to use the corresponding company’s mail service (either Live Mail/Hotmail or GMail), to host the users in your own domain.

I’ve been doing something similar to this for my own email at my domain, by setting up my accounts to forward messages to Windows Live Mail or Gmail (depending on the account). The problem here is that:

People you’re sending mail to can figure out where the mail comes from (Hotmail/Live Mail uses Reply-To, Gmail sets the From: address but uses a Sender: address of Gmail that will show up in some mail clients as “on behalf of”).
Only effective for your account - not everyone in the domain.

I credit Windows Live Custom Domains as having the first incarnation of the service. Basically, you tell the DNS system that the mail server (technically, this is changing your MX record) is not your hosting provider’s mail server, but rather it’s Google’s or Microsoft’s mail server. Some issues with any service like this:

There’s no halfway - it’s either everyone in the domain or noone
It’s hard to change back

There’s another issue that both services also have, at least right now, which is the ability to pull in all your existing mail. Of course, I believe both Hotmail and Gmail have the ability to pull from POP, so there is at least that migration path.

For the tests, I registered a free .info domain (lerchweb.info) at 1&1, a large hosting provider. I signed up for the Google Beta and the Windows Live Custom Domains beta (internally this is known as BYOD for Bring Your Own Domain). They both work essentially the same, with the following differences:

Windows Live requires that each user establish a Hotmail/Windows Live account. So far (and this is preliminary), Google does not require this. I would give Google the edge on this point, but Windows Live may have a better privacy story to tell.
Google lets you brand GMail with your own domain logo. Very cool touch.
Google lets you import users from a spreadsheet (exported to CSV). Also very cool, and this is a new feature that Windows Live may replicate.
Google beta took me about a month to become a member. Windows Live Custom Domains was immediate.

It’s hard to ignore my feelings about the actual mail system (I tend to prefer Google), but judging on simply the domain hosting features, I think I give this one to Windows Live Custom Domains, primarily because it takes so long to get into Google’s beta.

Next step is to get another free .info domain so I’m not repointing things all the time so I can do more than a 24 hour evaluation. This concept excites me, and I look forward to some cutthroat competition!

Slow day

Thu, 30 Mar 2006 10:27:00 -0800

This is one of those very rare events for me - a slow day. This day has been brought to me by a a series of events that have all coincided at once:

India has a holiday, so development and QA teams are not working
Most of my projects are in QA right now
A major event is going to occur next Monday at work, and in order for me to finish preparing I need to get information from some coworkers
All systems are running fairly well
Projects have been completing, but fewer new projects have been created over the last 3-4 weeks, so overall workload has been light the last 2 weeks, allowing me time to catch up

Weird - I don’t know what to do with myself. I’m definitely not complaining…

Gmail vs. Windows Live Mail vs. Yahoo Mail

Wed, 22 Mar 2006 20:57:00 -0800

So I’ve been working in several email systems comparing and contrasting the features. Here’s my take:

Yahoo! Mail: I have a Yahoo! id, but did not set up an email account originally. I tried to sign up to test things out for this blog entry, but I get an error message. I dutifully reported the issue to Yahoo support (issue # KMM58274977V28957L0KM), but have not had any updates in the past month. I don’t think I should be forced to get a new Yahoo! ID, so they lose.

GMail: Technically a beta product, it’s pretty open to people. If anyone wants a GMail ID, let me know. I have most of my lerch.org forwarded here. I don’t use the mail search all that often, but I like how the filters use the standard search terms. The filters are very flexible, and it’s nice to be able to tag items with multiple tags (basically the equivalent of putting a message into multiple folders simultaneously). It’s very “web-like”, and the ecosystem is also nice. GMail notifier, browser plugins, the ability to see mail on Google.com’s personalized home page and POP access all make the system very accessible. On the downside, visiting GMail with two browsers simultaneously really slows the system down.

Windows Live Mail (aka Kahuna): I’m now using “M2” of the beta product for all my commercial mail from lerch.org. The idea was originally that it would be the account that would collect spam, but it turned into a bit more than that. Anyway, it’s basically a trimmed-down version of OWA (Outlook Web Access), with an Outlook look and feel. If you like that, you’ll like this, and you can get automatic notifications through MS Messenger. Also, rules that you define have priorities, so if two rules match the same message, you can define which one should apply.

While Windows Live Mail is very much like Outlook, which should be comfortable for a lot of people. It does not have POP access, and it doesn’t have the ability to keep a copy of a message in multiple folders. Also, it does not support all browsers (connecting with Firefox downlevels to a Hotmail interface, although as a rapidly evolving beta product Windows Live Mail may address this issue shortly).

Right now, I do like both, Live Mail primarily for the very polished interface, GMail for the ecosystem support and filter behavior. I seem to enjoy GMail better, and it’s nice to have POP so I can truly archive my mail if I ever want to move away from the system. I look forward to both systems moving out of beta.

Now, what I find really interesting is Windows Live Domains and the fast follower GMail BYOD (Bring Your Own Domain) service. More to come on this…

Why I love STX

Thu, 16 Mar 2006 14:20:00 -0800

Seagate = Hard Drives
Hard Drives…needed for 1st Gen PVRs (e.g. Tivo, Windows Media Center) and future (High Def) generations
Hard Drives…needed for personal music players that have lots of storage (or who add video)
Hard Drives…needed for advanced gaming machines
Hard Drives…needed for all the music and/or videos you’re downloading
Hard Drives…needed for all those 5MP digital pictures you’re taking

Advanced gaming machines, personal music players, digital photography, peer-to-peer downloading, PVRs…all are examples of technologies that are expanding rapidly and are at various (leading) points of the technology adoption curve. These technologies are using more and more data, and the technology is quickly becoming mainstream.

The data storage industry is dominated by EMC, but EMC is mostly a business storage deal. Other major consumer data storage businesses are Western Digital and Maxtor. Seagate is buying Maxtor, and is cheaper (in terms of P/E) than Western Digital (in my opinion, for no good reason).

Bottom line is that this company is doing the right things, positioned well in an expanding industry, and should enjoy higher than average shareholder returns for the next several years. Major threats? Western digital drives prices down and kills the industry, but they can still lower margins and increase turns in an industry that’s expanding so rapidly. My biggest concern would be optical/holographic technology that provides much higher capacities in the same size form factor, but I believe we’re still several years out from that, giving Seagate an opportunity to develop the technology themselves or purchase an emerging player in that market.

Continuous Integration

Sat, 11 Mar 2006 15:38:00 -0800

I’m a big fan of the concept of continuous integration, which is the concept that code is built and tested as early and as often as possible. As soon as a change is completed/checked in, the code is built and tested. Usually this is done automatically (as an aside, I wonder how this approach can work well with enterprise software). Anyway, I saw this article and I found it pretty amusing. Now I have a good reason to get a rubber chicken…

Travel tip

Fri, 10 Mar 2006 14:18:00 -0800

Just got back from Taiwan a few hours ago, and I’m refining my theory about flying business class. Here are the two rules that I have now:

If flying for business and the flight is >6 hours, your company should pick up business class (Intel does not, and I think the policy is ridiculous - the difference in how rested you are at the end of a long flight between business and economy is crazy)
If you’re going to upgrade yourself due to poor company policy, try to get upgraded on the way back, rather than on the way out. This may be a work/life priority thing, but for me I’d much rather push through some business meetings than be exhausted when I get home and want to do things with friends and family.

Unfortunately, I did not get upgraded (was waitlisted but didn’t clear the list) on the way back home, so I’m currently exhausted. Off to bed for a couple hours…

Quick bit about Apple and Sony

Sun, 05 Mar 2006 09:16:00 -0800

I feel much the same way about Sony’s products as I do about Apple. I will never, for instance, buy any product that takes a memory stick.

However, there’s a big difference in my mind between the two companies. First, Sony appears to be (slowly) realizing the error of it’s proprietary ways. Second, as an investor, Sony is a huge conglomerate with revenue sources from many areas. While the second fact actually slows down adoption of open standards, it does smooth out the revenue stream.

There’s quite a bit I could also discuss about Blu-Ray here, and while blu-ray has its drawbacks, it is (at least in theory) an open standard.

Word to the wise on blogspot.com

Thu, 02 Mar 2006 08:35:00 -0800

Umm…apparently it’s bad news to use the < character flippantly on Blogger. My last post talked about AAPL P/S being &lt4, and when used in compose mode, it had the effect of mangling my last paragraph. I’ve actually been generally underwhelmed by Blogger’s inability to properly format without forcing me to use HTML. Users should not have to necessarily understand and manipulate the underlying technology.

Don’t confuse that statement by thinking it means systems should hide the underlying technology completely - power users should have access. My point is that general users shouldn’t have paragraphs mangled when they type in a < sign, and have to:

Know what broke
Go to HTML to fix it
Know the HTML code for <

Much Ado about Apple

Thu, 02 Mar 2006 08:02:00 -0800

Working for Intel, I’ve been hearing even more than the average person about Apple and how great it is. There are a lot of great things about Apple - here’s a few:

Great product design: I don’t know anyone who doubts or questions Apple’s product design prowess. This involves both hardware (iPod, iMac) and software (OSX).
Steve Jobs: Steve definitely has a way about him. He is a great leader; however, in my mind not necessarily a good manager.
Recent stock price: A year ago, AAPL was trading just over $40/share. It made it over $80 before settling back to it’s current price of $69.35 (10:55AM EST March 2nd, 2006).

Please, stop the insanity. Apple will always be a player, and never be the player, at least not for very long, at least not while Steve Jobs is running the ship. Why? Despite 20 years of evidence to the contrary, Steve still seems to believe that proprietary systems, vertical integration and customer lock-in are the way to profitability, market share and investor value. When they hit on a great product, the stock will go up. When they can no longer innovate the product to keep its popularity up, the fad will go away, and Apple will lose revenues, profit, market share, and stock value. I see no evidence to suggest a sustainable model.

Sometimes I look at a stock and am completely at a loss to explain how the market can value the company the way it does. In Apple’s case, P/S right now is &lt4, which does a fair job of explaining it, despite a 37.48 ttm P/E and a 26.73 forward P/E. However, Apple’s company and stock, in my opinion, is a house of cards, and I’ll stay away from AAPL, despite a belief that in the short run, there are profits to be had. Oh, and my music player? It’s a Creative MuVo TX FM (1GB). I’ll write about the virtues of it in another post.

Wow - last post in 2004

Wed, 01 Mar 2006 10:45:00 -0800

So, with energies poured into other things, it’s been a very, very long time. My intention from here out is to be a little more regular (I’m shooting for at least once/yr). ;-)

I’ll come back over the next couple days to talk about the baby, ASP.Net 2.0, the new car, phone/mp3 player, etc., etc. Hopefully with the baby site I won’t be posting to a crowd of no one!

With the baby I have finished up doing an application for our contest. Couple factoids about the site/technology/industry:

.NET hosting providers are consistently more expensive, especially if you want a database. I’m sure this is due to an MS tax, but also makes me wonder about TCO for .NET.
ASP.Net 2.0, which really does do a good job of killing off most of the hand-coding that used to be needed. Typical Microsoft evolution means that figuring out/controlling what it’s doing is more difficult if you don’t want the packaged behavior.
My phone has a web browser, which I will need when the baby is born and I need to call people. One of the most difficult/time consuming parts of the site was getting the infrastructure to understand that my phone can accept cookies. Not absolutely necessary, but I did not want to have to log in (which is necessary for the admin interface) during all the excitement.

Emil Lerch

wttr.in

wttr.in

Installing

A hard blocker

5 years go by

Starting out

First pass (6hrs)

Refining (15hrs)

Finishing up (1 week)

Conclusions and future work

Building an offline, limited version of Alexa

Building an offline, limited version of Alexa

(or how Belkin and Alexa conspired to nerd snipe me)

Conspiracy Theories Start Here!

Time to migrate

Investigations

Replacing Alexa for device control

Step 1: Speech to Text

Step 2: Understanding the text

Shipping

How I work

How I work

Overview

How does this work? Part 1: Bootstrap and configuration

How does this work? Part 2: Working with shells

How does this work? Part 3: Getting my normal programs onboard

How does this work? Part 4: Software development

Overall

Knack backpack

Knack Series 2 Medium Expandable Knack Pack

The Good

The Bad

The still unknown

Building Unikraft unikernels the hard way

Building Unikraft unikernels the hard way

Getting started

Hello world!

What the heck just happened?

Upgrading the base code

I haven’t done anything cool yet!

Why I can has no cheezburger?

I really can’t write to the console? How is helloworld.c doing this?

Loose ends

unikernels

Unikernels

What is a Unikernel?

Current Landscape

Enter Docker

Unikraft

Neomutt setup

Neomutt

SMTP and email notes

SMTP and email notes

Security

Software

Advanced terminology and use cases

Tools

Results

Zig build, and how I created a "universal lambda" function

zig build overview for zig 0.11.0, and how I created a “universal lambda” function

Terminology

What happens at zig build?

Step 1: Package fetch

Step 1: Package fetch…the big lie

Step 2: Compiling the build executable

Step 3: Running the build executable.

An example: A “universal lambda function”

Console application and WASM/WASI

AWS Lambda

Flexilib

Cloudflare

Conclusion

Wrapping up our exploration: A mini shell

Wrapping up our exploration: A mini shell

Learning the SDK and USB protocol

Learning the SDK and USB protocol

Getting two interfaces

Making both devices functional

Simplifying our tool chain: Wrap up

`zig build` overview for zig 0.11.0, and how I created a “universal lambda” function

What happens at `zig build`?

Step 2: Compiling the `build` executable

Step 3: Running the `build` executable.