Falvotech.
Conquering the world is easy — what do you do with it afterwards?

L I N K S


⇐ Finding Call-Sites After Backward Incompatible Interface Changes

⇒ On Forth Patterns and Practices


Unsuitable GOS Expanded to 256KiB
Samuel A. Falvo II
kc5tja -at- arrl.net
2010 Feb 11 23:02 PDT

Unsuitable's General Object Store (GOS), used to hold all the textual content of this blog, is running out of room to hold future content. For development purposes, it was specified only for 64KiB of space, of which only 8.8KiB remained as this article was written. To handle this problem, I expanded the GOS to 256KiB. This should give me sufficient room for an estimated 68 more articles before I have to resize the GOS again. A cheat-sheet for how I administered Unsuitable follows.

1 Relocating the GOS

Prior to the GOS re-allocation, the mappings.fs file contained the following:

: >core     dup 10 rshift block swap 1023 and + ;
: blocks    1024 * ;
: @f        >core @ ;
: !f        >core ! update ;

( block 1 = meta-block )
: g-fencepost   1 block ;
: a-nextId      1 block [ 1 cells ] literal + ;

( blocks 2 .. 65 = general object store )
2 blocks constant gorg
gorg 64 blocks + constant gend

( blocks 66 .. 67 = handle table for general object store )
1 blocks constant /hfields
66 blocks constant addrs
addrs /hfields + constant lens

( blocks 68 .. 77 = article table columns )
2 blocks constant /afields
68 blocks constant articleIds
articleIds /afields + constant titles
titles /afields + constant leads
leads /afields + constant bodies
bodies /afields + constant timestamps

Note that only 64 blocks served as GOS' backing store. To grow the GOS, I needed a temporary allocation, so that data migration could occur, thus giving rise to the new copy of mappings.fs:

: >core     dup 10 rshift block swap 1023 and + ;
: blocks    1024 * ;
: @f        >core @ ;
: !f        >core ! update ;

( block 1 = meta-block )
: g-fencepost   1 block ;
: a-nextId      1 block [ 1 cells ] literal + ;

( blocks 2 .. 65 = 64K unused space -- formerly general object store )

( blocks 66 .. 67 = handle table for general object store )
1 blocks constant /hfields
66 blocks constant addrs
addrs /hfields + constant lens

( blocks 68 .. 77 = article table columns )
2 blocks constant /afields
68 blocks constant articleIds
articleIds /afields + constant titles
titles /afields + constant leads
leads /afields + constant bodies
bodies /afields + constant timestamps

( blocks 256 .. 511 = General Object Store [256K version] )
256 blocks constant gorg
gorg 256 blocks + constant gend

2 Data Migration

I created the following code inside the util.fs file — it works by copying 64 blocks from one location (specified by f) to another (specified by t). The variables represent from and to, respectively. I use bf as a temporary holding buffer. While not efficient, it proves most effective.

variable    f
variable    t
create      bf  1024 allot
: b         f @ block bf 1024 move  bf t @ block 1024 move update  1 f +! 1 t +! ;
: 8b        b b b b b b b b ;
: movgos    2 f !  256 t !  8b 8b 8b 8b 8b 8b 8b 8b ;
movgos

After the above code executed, I removed the definition of movgos, since it wasn't needed anymore.

3 Fixing Up Handles

The article table (defined by the articleIds, titles, leads, bodies, and timestamps columns in mappings.fs, above) references strings by way of handles. I knew, from the get-go, I'd eventually have to relocate and resize the GOS, so choosing to express strings through opaque handles proved an excellent choice, for then all I'd need to do is update the addrs column in the GOS handle table. The following code, executed interactively, proved sufficient to do the job:

: ra   addrs begin dup @f -1 xor while dup @f 256 blocks + over !f cell+ repeat drop ;
ra

After the invokation of ra (relocate addresses) completed, the Unsuitable database regained consistency, and is now safe to accept new article submissions, of which this is the first one since relocation and expansion.

4 Conclusion

This article contains a cheat-sheet for instituting capacity-planning policy changes with Unsuitable. As you can see, it's quite easy to administer Unsuitable. No, there exists no fancy GUI. Yes, you need to think logically about what you want, your posting rates, and size your table accordingly. DBAs, Hadoop administrators, and so forth do this sort of thing all the time. I invested a grand total of 10 minutes into this effort, most of which were spent refamiliarizing myself with Unsuitable's otherwise undocumented code (though, hopefully, with time, I aim to change that).