Edit: Due to a high troll rate on this post, I've temporarily disabled commenting.  If you'd like to leave feedback, please email me at cody.brocious@gmail.com or drop a comment on the Hacker News thread http://news.ycombinator.com/item?id=1254678 .

I'm proud to announce the first beta of Dotpack, a packer for .NET executables. I've been working on this for the last week or so and finally have something fit for public eyes. It started out of my desire to build 64k demos on .NET and for that reason it's very small -- as of this writing, it's sitting at 5331 bytes overhead. It also achieves high compression ratios due to its use of LZMA; average size reduction for the files I've tested has been 60-80%.

At the moment it's fairly straightforward, not tampering with the original binary at all, but future versions will bring obfuscation and an array of code transformations to drop the filesize even further.

This version has several known issues I simply didn't have time to deal with: Silverlight packing is there (you can pass it a .xap and get one back), but it's very finicky and not particularly good so far. Packing binaries that use System.Reflection.Assembly.GetExecutingAssembly().Name will get back an empty string due to the way assemblies are loaded after unpacking, which can cause major issues. I'm going to fix these for the next release.

Other future features which will be coming, in no particular order:

• Merging of assemblies. I was originally planning on releasing with ILmerge support as a stopgap until I finished my own prelinker, but licensing issues and a generally poor API made that less appealing. I'm working on a prelinker which, in addition to just merging assemblies, will perform dead code analysis to strip unused portions of the code away.
• Obfuscation. Not only will this make it more difficult to analyze your binaries, but you'll get the benefit of less space being taken up by names. This can be quite substantial in a large binary.
• Visual Studio integration. You'll be able to easily tie Dotpack into your Visual Studio workflow to produce packed binaries from your release builds.

Dotpack is freely distributable, but is under a non-commercial license. If you're using it in a commercial environment, even if you're not distributing your binaries, please purchase a commercial license. In addition to supporting Dotpack's development, you will also get builds ahead of the non-commercial users.

You can get the current beta build of Dotpack here: http://straylight-software.s3.amazonaws.com/Dotpack1.0b1.zip. To use it, simply run: dotpack.exe input.exe/.xap output.exe/.xap and you're off.

During the beta, a one-year single-user license is available at a discounted price of $250; once this goes gold, the price will go up to $500. Note that the year doesn't begin until the final release, so you're getting quite a deal. If you'd like to purchase a volume license, please contact me at cody.brocious@gmail.com. (Before the final release, I plan on getting a site up, but you know what they say about minimum viable products.)

Try it out, let me know how it works for you, and let me know if there are any features you'd like to see.

Happy Hacking,
- Cody Brocious (Daeken)

Loading mentions Retweet

Well, this demo is coming a bit late. After days of fighting with OpenGL and GLSL, I finally have a demo that's fairly presentable. It's still not great (by any means), but it's an improvement over last week's demo and that's the goal. Sadly, this week's demo has compatibility issues out the ass -- I suddenly understand why everyone goes D3D for the demoscene. My next demo will almost definitely be D3D10, due to familiarity and geometry shaders. We'll see.

Anyway, you can get the build at http://pouet.net/prod.php?which=54524. If you give it a shot, let me know what video card you're using and whether it worked or not. I'll be releasing the source in the next day or two if anyone wants to play with it, and as always, the NFO is below.

Oh, and note: I couldn't make it less dependent on framerate this time around, but it's a ton faster than the last demo, so I'm not going to beat myself up over it. Next demo will be framerate-independent, I promise...

Happy Hacking,
- Cody Brocious (Daeken)

                 Armitage by Straylight
                    - April 2, 2010

Week 2 of my demo-a-week project is upon us.  I tend to think this is a much
better demo than the last one, even if it is short.  It has known compatibility
issues (older ATI cards, like the one in my desktop, seem to dislike my vertex
shaders...) and it'ss fairly short, but I like it a lot more than Waveride.

I was planning on making this one cross-platform, but after dealing with tons
of compatibility issues, I figure it's better for my mental health if I didn't.
The source will be made available in case anyone feels like doing something fun
with it.

Waveride used Miriel by Nightbeat as its music, on suggestion from a friend of 
mine.  Oddly enough, Nightbeat produced a song called Straylight -- I got the 
name from Neuromancer, but I can only take that as a good sign.  Anyway, enough
rambling.

Compatibility:
Tested on a GeForce 9400M, but should work with most modern cards.  Requires
SM3.0 (I believe) and rendering to a floating point framebuffer object.  If it
dies on start or you don't see the pretty spheres, one of those things is 
likely the issue.

Credits:
Code and art by Daeken
Music by Novoxide

Greets to:
ASD
SVatG
Conspiracy
Fairlight
Kewlers

Until next time,
- Daeken

Loading mentions Retweet

In the course of working on my next demo for my demo-a-week project, I'm doing a lot of procedural texture generation. The ability to quickly test modifications to my algorithms is crucial, so I decided to whip up a simple texture editor, Straytex (named after Straylight, my demogroup).

It likely only works on the latest Chrome dev channel build (edit: also works on Safari and Firefox, but I'm not sure which versions) and is ugly as sin (if anyone wants to help out with either of these issues, it'd be greatly appreciated!), but it gets the job done. At the moment, the interface is pretty simple: you can select the texture size, change the random seed, modify the source, and tag your changes. The tag feature is especially handy -- you often accidentally stumble onto cool effects, so the ability to tag these to go back to them later is insanely useful.

You can see it live here, and check the source out on Github. I hope you enjoy it.

Happy Hacking,
- Cody Brocious (Daeken)

Loading mentions Retweet

For a long time, I've sat on the sidelines of the demoscene, generally watching in awe as ASD, Farbrausch, Conspiracy, and others produce amazing works of art. I recently decided that I'd jump in and see what I can do, and to accomplish that I've decided to start releasing a demo a week. My goal is to get my feet wet and learn how to actually build something beautiful.

Today I'm releasing my very first real demo ever, Waveride, which I decided to hack together over the course of the last few hours. There's not much to it -- a wave plane consisting of spheres and droplets falling onto it with locations based on the music data -- but I don't think it's bad for a first shot. I'm planning on improving it over the coming weeks, before moving on to something new. At the moment, it's Windows-only, but pure OpenGL -- it should work fine in Wine. I'm going to port it to OS X and Linux before moving on.

Anyway, here's a link if you want to check it out in its current incarnation: http://pouet.net/prod.php?which=54508. I've already received a good bit of information on what I need to fix (the nauseating rotation is absolutely first up on the list, I swear!), but please give me anything you can think of. This is an experiment for growth, and I can't grow if I don't know what I need to improve upon. The NFO is below.

Happy Hacking,
- Cody Brocious (Daeken)

                Waveride by Straylight
                - March 25, 2010

Welp, not much to say.  I said I was going to create a demo a week until I had
a clue what I was doing.  Waveride is the first of these demos, and I'm pretty 
damn proud of it.  It's not perfect, and I'm not terribly happy with having 
used an off-the-shelf song for it, but eh.

Decided to sit down and write a demo, and I did it.  That was 1am, this is 8am.
Can't complain too much.

Developed and designed by Daeken.
Music is Miriel by Nightbeat (I believe this is cool -- if not, please, please 
let me know, and I'll change it at once.)

Greets to:
Nightbeat
ASD
SVatG
Lateralus
Kewlers

Until next time,
- Daeken

Loading mentions Retweet

(Edit: The Daeken Discount Program is closed.  While an interesting exercise, it resulted in only one sale, but it was of the 25% discount level so I can't complain too heavily.  If nothing else, it was fun to try something.)

A long while back, I saw people selling shares in themselves. That is, you invest in the person and get a portion of their earnings and such, just like investing in a company. While I like the idea in theory, I can't see selling off a large (1+%) part of myself that I'll likely never get back. However, the idea's stuck with me. So while I was laying in bed last night, I came up with an idea.

The Daeken Discount Program

I'm launching an experiment. As of today, you can buy a lifetime discount on any of my products; rules and restrictions are below.

• $50 gets you a 5% discount
• $90 gets you a 10% discount
• $200 gets you a 25% discount

I see this as a win-win. If no one buys into it, I'm out an hour worth of work and write this off as an interesting failure. If people do buy into it, I potentially lose some future profits, but the people who would do so are going to be more likely to buy from me anyway.

Why should anyone buy into this?

I think that people will see that I build cool things, and it's a small enough amount of money that everyone that buys into it will eventually at least break even.

How does it work?

The discount will be tied to the email you put in the Paypal purchase (although if you end up changing emails, I'll move it over to another, of course), and when you use it for purchases on anything of mine, you'll immediately receive the discount.

What does it apply to?

Any product purchase of mine, whether one-time or recurring. The obvious exception is if I don't have the right to do this, e.g. a product I build for someone else. However, all of my future companies will fall under this.

How long will you run this?

Probably no more so than a month or so. If it succeeds, I'd rather not have a million people with lifetime discounts on my products; if it fails, there's no point in leaving it up.

What have you built and what are you building?

Product-wise, I haven't built much; the biggest thing I've built was Alky, which allowed the conversion of Win32 binaries to run natively on OS X and Linux. It was a marginal success, although it ended up failing later for business reasons.

These days, I'm working on a few things:

• Renraku: http://daeken.com/renraku-future-os This will eventually be combined with
• OpenBAMF/IREctive: Reverse-engineering platform and module store (My primary for-profit project right now)
• Books
  • The Emulator's Handbook: A book on building an emulator from start to finish. There's simply nothing there yet, which is a damn shame -- we need to get people involved here.
  • So far unnamed: A book on reverse-engineering game protocols and emulating them.

In the future, I'd like to be developing and selling Eyetaps and other hardware around Renraku.

What if you never build anything interesting? / What if you never start another company?

In the (I hope, unlikely) event of one of these happening, I'm deeply sorry. I do my best to create things that people will want, but it's entirely possible that I'll fail. If I end up working a day-job for the rest of my life and someone buys into this program, I'll do whatever is in my power to give you the discount, even if I have to pay the difference myself.

Can I buy it multiple times?

No, sorry, but this does apply on top of any other discounts available.

What if you renege on the deal?

Then I'd hope that I'd be outted as a fraud and prevented from ever doing business again. Seriously, I wouldn't do business with someone who pulled something like that, and I'd sincerely hope that others wouldn't either.

Questions?

Post a comment or email me at cody.brocious@gmail.com. I'm curious to see whether or not this takes off; would be cool for it to do so.

Happy Hacking,
- Cody Brocious (Daeken)

Loading mentions Retweet

Earlier this week, I had to deal with some files in Python's marshal format (some .pyc files, specifically) in Ruby and discovered that the details of this format aren't documented. Since it's meant to be purely internal, the Python team has decided not to document it in any way.

The marshal format is used in .pyc files, lots of internal storage for random apps, etc. It's a shame that it's undocumented, as this means that there are, to my knowledge, no implementations for other languages. This also means that if you have a malicious marshal blob, you have to load it up with Python to play around with it; not a good idea.

Fortunately, you can read the source (Python/marshal.c) and figure out how it works pretty easily. However, to make it even easier, I've decided to write up some simple documentation on the format. I'll give types as int/uint<n> where n is the number of bits. The object type indicates that this is a marshalled object.

Format

The marshal format in and of itself is very simple. It consists of a series of nested objects, represented by a type (uint8 -- a char, in fact) followed by some serialized data. All data is little-endian.

Note: I wrote all of this for the 2.x line. I don't know how much has changed in 3.x.

Types

Constants

These types contain no data and are simply representations of Python constants.

• 0 (TYPE_NULL) -- Used to null terminate dictionaries and to represent the serialization of a null object internally (not sure if this can happen or not).
• N (TYPE_NONE) -- Represents the None object.
• F (TYPE_FALSE) -- Represents the False object.
• T (TYPE_TRUE) -- Represents the True object.
• S (TYPE_STOPITER) -- Represents the StopIteration exception object.
• . (TYPE_ELLIPSIS) -- Represents the Ellipsis object.

Numbers

• i (TYPE_INT) -- Represents a int on a 32-bit machine. Stored as an int32.
• I (TYPE_INT64) -- Represents a int on a 64-bit machine. Stored as an int64. When read on a 32-bit machine, this may automatically become a long (if it's above 2**31).
• f (TYPE_FLOAT) -- Represents a float in the old (< 1) marshal format. Stored as a string with a uint8 before it indicating the size.
• g (TYPE_BINARY_FLOAT) -- Represents a float in the new marshal format. Stored as a float64. (Thanks to Trevor Blackwell for noting that these are not float32 (along with TYPE_BINARY_COMPLEX).)
• x (TYPE_COMPLEX) -- Represents a complex in the old (< 1) marshal format. Contains the real and imaginary components stored like TYPE_FLOAT; that is, as strings.
• y (TYPE_BINARY_COMPLEX) -- Represents a complex in the new marshal format. Stored as two float64s representing the real and imaginary components.
• l (TYPE_LONG) -- Represents a long. Haven't yet figured out how this works; I'll update shortly with that.

Strings

• s (TYPE_STRING) -- Represents a str. Stored as a int32 representing the size, followed by that many bytes.
• t (TYPE_INTERNED) -- Represents a str. Identical to TYPE_STRING, with the exception that it's added to an "interned" list as well.
• R (TYPE_STRINGREF) -- Represents a str. Stored as a int32 reference into the interned list mentioned above. Note that this is zero-indexed.
• u (TYPE_UNICODE) -- Represents a unicode. Stored as a int32 representing the size, followed by that many bytes. This is always UTF-8.

Collections

• ( (TYPE_TUPLE) -- Represents a tuple. Stored as a int32 followed by that many objects, which are marshalled as well.
• [ (TYPE_LIST) -- Represents a list. Stored identically to TYPE_TUPLE.
• { (TYPE_DICT) -- Represents a dict. Stored as a series of marshalled key-value pairs. At the end of the dict, you'll have a "key" that consists of a TYPE_NULL; there's no value following it.
• < (TYPE_SET) -- Represents a set. Stored identically to TYPE_TUPLE.
• > (TYPE_FROZENSET) -- Represents a frozenset. Stored identically to TYPE_TUPLE.

Code objects

Code objects (like that in a .pyc file, or in the func_code property of a function) use the c (TYPE_CODE) type flag. Even in the case of the top level (as in a .pyc), they represent a function.

They consist of the following fields:

• argcount (int32) -- Number of arguments.
• nlocals (int32) -- Number of local variables.
• stacksize (int32) -- Max stack depth used.
• flags (int32) -- Flags for the function.
  • 0x04 -- Has *args.
  • 0x08 -- Has **kwargs.
  • 0x20 -- Generator.
  • This list is not all encompassing; certain __future__ declarations will set their own flags.
• code (object) -- String representation of the bytecode.
• consts (object) -- Tuple of constants used.
• names (object) -- Tuple of names.
• varnames (object) -- Tuple of variable names (this includes arguments and locals).
• freevars (object) -- Tuple of "free" variables. (Can anyone clarify this a bit?)
• cellvars (object) -- Tuple of variables used in nested functions.
• filename (object) -- String containing the original filename this code object was generated from.
• name (object) -- Name of the function. If it's the top level code object in a .pyc, this will be <module>.
• firstlineno (int32) -- First line number of the code this code object was generated from.
• lnotab (object) -- String mapping bytecode offsets to line numbers. Haven't delved into the details here.

RMarshal

I've implemented support for unmarshalling objects as well as reading .pycs in a Ruby gem called RMarshal. You can get it from Gemcutter or from Github.

Closing

Hopefully this will be of use to someone. One potential use is in studying malicious marshalled data; the Python guys strongly recommend against unmarshalling untrusted data, but we all know how well such notices are regarded. In addition, it may help you manipulate Python bytecode from non-Python languages.

Drop me a line if you do anything cool with it.

Happy Hacking,
- Cody Brocious (Daeken)

Loading mentions Retweet

Renraku has seen a renewed surge of energy in the past few weeks, and has made big strides. Here's what's going on.

Where we are

Renraku now runs in hosted mode on top of MS.NET/Mono. This enables a much, much quicker development/debugging cycle, and allows us to start working on the GUI while the low level details are still being worked out. There's still work to be done here, but it's a good start. We've also made the leap to Rake from NAnt; dealing with the bulky, inflexible build files just got to be too much. A few hours of hacking and Rake now fits our purposes perfectly.

After hosted mode came up, we made a push to get video/mouse services built on top of SDL, enabling us to start developing the GUI. You can see an initial GUI coming up and displaying our logo here:

In addition, Capsules (check out this blog post if you don't know what this is: Renraku OS: Networking, Hosted Mode, Moving Forward) are in progress and are taking shape; the implementation is still rapidly changing, but we seem to have a solid idea of what they're supposed to look like and how they interact with each other.

Where we're going

The project is now moving in parallel on two efforts: Userspace (GUI, applications, storage), and native (kernel, compiler, drivers). This will all converge over the next 6 months.

We're aiming to make two releases between now and July 4th, which will serve to get the code in the hands of developers leading up to v1.0. We now have a solid roadmap for Renraku v1.0, which we aim to release on July 4th, the one-year aniversary of the project. The roadmap is as follows:

Goal

The project goal is that v1.0 should go out on July 4, 2010. This marks the first anniversary of the project and will be the first major release.

What should it do?

You should be able to start Renraku in hosted or native mode (IA32 native only) and bring up a usable GUI. You should be able to browse files, start applications, edit configuration, and run UI tests. Depending on time constraints, we may or may not have additional applications (and a game?)

What needs to be done?

Compiler:

• Exceptions
• Generics
• Array bounds checking
• Emit proper class data for reflection

Gui:

• Basic toolkit for controls
• Image rendering
• Font rendering
• Vector support?

Graphical shell:

• Window management
• Desktop icons
• Some sort of system tray/menu

Kernel:

• Tracebacks on exceptions
• Garbage collection
• Memory management
• Storage
  • Low level storage service (hard drive access)
  • FAT32 filesystem service
• Networking
  • TCP
  • IP and UDP supporting fragmenting
  • More robust DHCP
  • DNS
  • Routing
  • Transparent object remoting
• Video driver on IA32 (better than VGA)

Services:

• Capsule implementation
• Service documentation

BCL:

• Migrate to Mono BCL
• Implement more of the BCL, if migrating to the Mono BCL isn't possible/practical

Applications:

• GUI console
• File browser
• Basic text editor
• Image viewer
• Some basic GUI game?

Build System:

• Allow portions of the code to be tagged as platform-specific, rather than the large file lists in the Rakefile
• Integrate building a LiveCD with a complete Renraku system.
• Build Renraku installer (XXX: May get pushed back to v2)

How you can get involved

At this point, we need a few things: Application developers, service developers, kernel developers, and compiler developers. Want to write the first Doom port to run on a managed OS, a Twitter client, a completely new web browser, or a sane networking stack? Here's your chance.

No matter how you want to help out, or if you just want to ask some questions, join us at #renraku on Freenode IRC.

Points of contact:

• Daeken (Cody Brocious, cody.brocious@reaversconsulting.com) -- Kernel lead; Kernel, services.
• ircubic (Daniel Bruce, ircubic@gmail.com) -- Userspace lead; GUI, applications, storage. Documentation lead. Build system.
• nrr (Nathaniel Reindl, nrr@corvidae.org) -- BCL lead. Memory management lead.
• dublindan (Daniel Kersten, dkersten@gmail.com) -- Capsule lead.

Happy Hacking,
- Cody Brocious (Daeken)

Loading mentions Retweet

A lot of people have asked me what happened to the Alky project. The short answer is that we made a lot of bad business moves, but that answer glances over a lot of the fun little details. Having gained considerable knowledge from other stories of failed startups, I figure I'll throw one of my own into the ring.

History

The Alky project's history can be split into a few phases:

Conception

Alky began as an experiment to see how easily I could convert Windows PE files to run natively on Mac OS X (x86). If it were to work, it may make it possible for me to convert Windows games to run natively on OS X, which was my primary focus. I started by writing a converter that ripped the segments out of the original file and throw them into a Mach-O binary, then linking it against 'libalky'.

LibAlky was a reimplementation of the Win32 API. At first, I tested with files that just called a few simple things, like user32:MessageBoxA, and it worked spectacularly. It was at this point that I realized the basic concept was not only sound, it made a whole lot of sense.

Actual project creation

Once the initial prototype worked, it was time to get people interested. I went to Michael Robertson (who was my employer at the time) and gave him a rundown. He offered to buy the domain, host the project, and get some resources behind it, primarily PR-wise. Within a few days, the project started actually feeling real. We got the site up, wrote some copy to explain what we were doing, and then put it out on Slashdot.

Unsurprisingly, we received three types of responses:

1. This is impossible, it simply can't work from a technical point of view. (This was especially hilarious coming from a Transitive engineer, considering that what they did is considerably more complicated.)
2. This is possible, but it won't happen due to the immense amount of work involved. (Half right -- more on this later.)
3. Wine sucks, I hope you guys can do it better. (We couldn't -- more on this later.)

But more importantly than anything else, we got some developers involved. However, they ended up being driven away.

Mismanagement of the open source project

Alky was the first open source project I'd ever managed that consisted of more than myself and a few friends, and as a result it was mismanaged at every possible turn. It was unclear what people should've been working on, no effort was made to document the project, and no real effort was made to include the developers that were so interested in working on the project.

This was compounded by a rewrite and redesign, which I decided (foolishly) to take on entirely by myself. Some of the design was done as a group, but none of it ever left my hard drive, so work stalled on the existing codebase and the project started to wither.

Shortly thereafter, Falling Leaf Systems was incorporated to back the development of Alky. This further increased the rift between the open source developers and the "core" group (consisting of myself and one of the cofounders of the company). Originally, we planned to dual-license the codebase, but as we got more into discussions of the goals of the business, it became clear that closing the source was the right move. However, we couldn't have picked a poorer way to do it.

Rather than be upfront about the move to closing the source, we simply killed the IRC channel and took down the site. The open source developers were left wondering what happened, while we moved on the rewrite.

Prey and the Sapling Program

Alky was completely rewritten with the new design, and work quickly moved forward on getting the first game running. We released a converter for the demo of Prey (Mac OS X only at first), as part of our new Sapling Program. The Sapling Program was created as a way to get some upfront money for the company, so we could get needed hardware, go to the GDC (which was a horrendous waste of money, for the record), etc. We sold memberships for $50, which gained access to the Prey converter and all future converters. Of course, after we finished Prey for Linux, there were no more converters.

Loss of focus

After Prey was done, we'd planned on implementing DirectX 9 with hopes of running Oblivion, but we lost sight of this goal. Instead, we decided to go after DirectX 10. Along with this shift in focus came an even bigger one: we were no longer targeting Mac OS X and Linux primarily, but Windows XP. We saw that Vista was tanking, and DirectX 10 was only available there, so we decided that we only had a limited window to make money off of that.

Shortly after we made the change, we released a library to allow a few of the DX10 SDK demos to run on Windows XP via OpenGL, albeit with serious missing features (e.g. no shaders). It got some attention, but few people were able to actually get it working. After this was out, I started work on reverse-engineering the shader format and writing a recompiler that would allow Direct3D 10 shaders to work with OpenGL, and even more importantly, without DX10-class hardware. Geometry shaders were planned to run on the CPU if the hardware wasn't available to handle it, and work progressed quickly.

Alky for Applications

We discovered a project known as VAIO to allow Vista applications (non-DX10) to run on Windows XP, and after some talks with the developers, they (or more specifically, one of them -- we'll call him Joe) were sucked into Falling Leaf. We rebranded VAIO and it was released as Alky for Applications. After this, Joe was tasked with making Halo 2 and Shadowrun -- Vista-exclusive but non-DX10 games -- run on Windows XP. We were so confident in our ability to do this, we set up an Amazon referral system and made it such that anyone who purchased either game through us would get a copy of the converter for free.

At this point, I was working heavily on DX10 stuff, and was under tight deadlines to show things off to a company that was interested in our technology, but the clock was ticking. About a week before the planned release of the Halo 2 and Shadowrun compatibility system, Joe came to us and told us that he'd not been able to get anything working, and had very little to show for the time spent. In retrospect, it was my fault for not checking up on him (my job as his manager), but at that point it just made me realize there was no way it was going to be done in time.

I picked it up -- dropping DX10 in the process -- and attempted, feebly, to get it done. Of course, I picked the most difficult way to approach it; reverse-engineering the Windows application compatibility system. By the time I got anything remotely close to working, we'd already missed our deadlines for both the DX10 work and the Halo 2/Shadowrun converter.

The death of Falling Leaf

After all this went down, I fell into a bit of a depression. I became demoralized to the point of just not wanting to go on anymore, in the face of repeated failures, very much in public. Despite us not walking away with a dime -- we made approximately $7000 in total, none of which went to any of the founders of the company -- I felt that we'd ripped people off, despite the best of intentions. It wasn't long after this that Brian (one of my co-founders) stepped down as CEO, and I closed the doors on the company. The source to Alky and Philosopher (the compiler framework used in the shader work) were released at the same time.

Lessons Learned

1. If you're going to run an open source project, make it easy for people to contribute. Not only that, make it worthwhile for them to contribute and make them a part of the project, not just free workers.
2. If you're going to kill an open source project, be up front with the people involved. It's not only dishonest not to do so, you lose people who may well go to bat for you even if you are commercial. This is especially important for a project like Alky, where we faced nearly universal negativity.
3. If you're going to change focus, be clear with your users as to what's going on, and make sure you make it clear that the old stuff is dead and gone. If you don't, you come off looking terrible in the press, and it just makes you look like amateurs (which we were).
4. Make sure your employees are actually doing what they're supposed to be doing, especially if they're working remotely. This was really the final nail in the coffin for Falling Leaf.

Alky Reborn

Now, with all of that said, there's a light at the end of the tunnel perhaps. The source for Alky has been pulled into Github and it seems that development is picking up again. Perhaps I can shed some light into what design decisions were made, how it was implemented, and how I'd do it now if I were so inclined. I don't plan on working on Alky again (that time has passed), but I'd still love to see it succeed.

The old Alky design

Alky's original prototype had a very simple design, library-wise. There was one big LibAlky which contained all of the Win32 API implementations, each talking directly to native APIs. This design very quickly became unworkable, as we had tons of duplicated, unmaintainable code.

The new Alky design

Alky was redesigned such that we had the Nucleus and the Win32 APIs. The Nucleus was intended to abstract away the platform-specific details and expose a universal API that the Win32 APIs could sit on cleanly. While a good idea, it quickly broke down in implementation. I ended up writing code that straddled the Nucleus and the Linux/OS X APIs, rather than abstracting everything away. This led to slower development and an even more complicated code base.

Potential new design

Having done two implementations of Alky and quite a few other projects that relate to it in concept (IronBabel, Renraku (OS design plays a factor here), etc), I think I'm at a place where I can perhaps come up with a workable design.

The key point where both Alky implementations (and Wine, IMHO) failed is in maintainability. The codebase was a rats nest, much like the real Win32 APIs, and neither implementation managed to help this at all. I think this needs to be the absolute top priority, above performance, compatibility, and all else. If your code is maintainable, all the rest can happen.

With that focus in mind, here are the things I'd do with a new design.

• Implement the APIs on top of Mono, taking advantage of the flexible marshalling that P/Invoke gives you. This will allow you to simplify things greatly, and will only have a marginal performance hit in most cases.
• In cases where performance is critical, drop down and implement things in C, C++, or assembly. If this chunk of the project is greater than 10% of the codebase, you've got bigger problems.
• Abstract, abstract, abstract. Break things into the smallest chunks possible and reuse them. This is what we tried to do with the Nucleus idea, but it was easy to just ignore it for complex pieces.
• Write thorough unit tests for every API that's implemented (public and internal). Regression testing would also make things really nice.
• Rather than trying to get real games/applications running immediately, write your own simple applications that test specific chunks. This would've cut down considerably on the development time in the old revisions of Alky.
• Write a great debugger to be able to step through applications cleanly. In the old days, I'd break out IDA Pro and step through a game on Windows, then compare the behavior to the Alkified version, but this was just downright painful.
• Make it work on Windows, to allow easy side-by-side comparisons.

The suggestion that this should be built on top of Mono/.NET sounds ridiculous, I'm sure, but I do think it'd give the project a shot.

In Closing

I hope this has given you some insight into what went down with Falling Leaf, some ideas of what not to do (obviously, it's easy to completely overlook these things when you're down in the trenches, as we did), and where Alky could one day go. I wish the Alky Reborn folks the best of luck, and I hope some of my advice helps.

Happy Hacking,
- Cody Brocious (Daeken)

Loading mentions Retweet

This is the current Renraku OS FAQ. If you have a question that's not answered here, feel free to join us in #renraku on irc.freenode.net.

What is Renraku OS?

Renraku is a pure managed code operating system, written in Boo from the ground up. The goal is to produce a high-quality operating system without the traditional security, performance, and flexibility issues.

Is Renraku open source?

Yes, Renraku is distributed under the CDDL (Common Development and Distribution License).

Why not use X in place of .NET?

The .NET infrastructure is fairly simple, powerful, (very) easy to compile, and flexible enough for our purposes. Another tool may be better suited to the task, but we've found it to be a good match.

Why use Boo instead of C#?

Although C# is the more traditional choice for .NET work, it lacks macros and is simply much more verbose. Boo is very easy to pick up and significantly more powerful. However, our choice of Boo doesn't mean that the rest of the OS has to be written in Boo. Services, Capsules, and drivers can all be written in any .NET language.

Is it POSIX compatible?

No, by design. POSIX was great in 1989, but times have changed and we need to move forward.

What about legacy applications?

Virtualization is the only planned route for legacy code at the moment, as it's the most efficient and lowest cost. Perhaps we'll support running certain types of legacy applications in a more direct route in the future, but it's doubtful.

Won't this be slow?

For a while, yes. However, compilers are rapidly getting considerably better and compilers like Microsoft's Bartok are paving the way toward faster code. In addition, our lack of separate address spaces and privilege levels make system calls and task switches dramatically cheaper than on existing OSes.

No separate address spaces? What keeps applications apart?

Because we use pure managed code, we're guaranteed that applications cannot touch memory they aren't directly given. This means that the compiler has the last word on all memory access in the code, reducing the attack surface of the OS considerably.

Does it run X?

Unless the application you have in mind is playing with our shell, looking at our logo, or acquiring an IP address via DHCP, probably not. Renraku is still in its infancy, but we hope it'll be useful someday.

Loading mentions Retweet

Developing Renraku has led me to some interesting thoughts, not the least of which is the feasibility of a .NET computer.

Concept

Existing computers know only a few core things:

• Move to/from memory and registers
• Branch
• Perform arithmetic

All the other smarts are built on top of that, leaving us with layer upon layer of complexity. What if we were to add some logic to the hardware and introduce a thin software layer on top of that?

Design

CPU

The CPU will execute raw CIL blobs and exposes hardware interfaces and interrupts. The most likely route is to keep the stack in a large on-die cache, doing a small lookahead in decoding to prevent multiple stack hits in arithmetic.

Since the CPU is going to be executing CIL directly, it's going to have to deal with object initialization, virtual method lookups, array allocation, memory manager, and many other tasks that existing systems handle purely in software. This is achieved by splitting the task between the CPU and a hypervisor.

Hypervisor

The hypervisor, like everything else, will be built completely in managed code. However, it will be a raw blob of CIL, instead of the normal PE container. The hypervisor will handle most of the high-level tasks in general, but it's up to the specific hardware manufacturer to determine what goes in the CPU and what goes in the hypervisor. The hypervisor will come up when the machine is booted, much like a BIOS. After performing hardware initialization, it will look for the bootloader.

The hypervisor provides a standardized interface to talk to drivers and memory, which are verified. In this interface, it will also provide hooks for the memory manager, allowing the OS to handle it in a nice way. Interrupts will also be hooked via this interface. The hypervisor instance will be passed to the bootloader, which then passes it on to the OS.

Bootloader

The bootloader is a standard .NET binary, and its job is simply to hand control off to the OS. Its function is effectively identical to that of a bootloader on an existing system.

OS

The OS will look very similar to existing managed OSes, with a few exceptions. Obviously, there's no need to write your own .NET implementation, and no need to write a compiler.

Because it exclusively talks through the hypervisor interface, it's possible to nest OSes as deep as you want, you simply have to expose your own mock hypervisor. You can handle resource arbitration here and run them very efficiently.

Unlike most OSes, it won't get free reign over the system. To access devices, it'll have to use verified channels with the hypervisor, to ensure that consistency is upheld. This is similar to the way that Singularity functions.

Is it a good idea?

Maybe, maybe not. I don't know how efficient a CIL chip would be or how much of the .NET framework can really be pushed onto a chip, but the core idea is really an extension of Renraku. A pure-managed system would be very powerful if done properly, but it's a pretty drastic change.

A nice middle ground would be utilizing an existing processor and using the hypervisor in place of the existing BIOS/EFI. With some code caching magic, the CIL->X compilation could be done at this level and be completely transparent to the OS. This may well be a better route than a CIL CPU, but it's tough without having more information on the hardware challenges involved with such a thing.

Happy hacking,
- Cody Brocious (Daeken)

Loading mentions Retweet