… I believe no other language can do that so short and still readable. 
Raku is so much fun to work with – I do not understand why it is not much more popular …
Raku RSS Feeds (individual feeds | subscribe to all via Atom)
Elizabeth Mattijsen (Libera: lizmat #raku) / 2026-04-22T11:25:45Post Image: Carolyn Emerick, CC BY-SA 3.0 https://creativecommons.org/licenses/by-sa/3.0, via Wikimedia Commons
Matt Doughty has served up a double helping of Selkie this week. This is a TUI (Terminal User Interface) module written in Raku that provides a simple, declarative way to roll your own TUI app in Raku. Please do check it out – I look forward to seeing a crop of Raku TUIs to feature in the weekly.
and
The pitch: you build a widget tree declaratively, mutate state, and the framework handles dirty tracking, rendering, focus cycling, resize, and teardown. Native performance without the pain.
Sizing uses a fixed/percent/flex model similar to CSS flexbox. There’s an optional re-frame style reactive store if you want centralized state with event dispatch, effect handlers, and path subscriptions.
Ships with a decent widget set out of the box: text inputs (single and multi-line), scroll views, list views, card lists, tables with sortable columns, tab bars, modals, progress bars, a command palette, file browser, image display via pixel blitter, and theming throughout.
Seven example apps in the repo covering everything from a minimal counter to a tabbed dashboard.
Also includes a testing toolkit (keystroke synthesis, Supply observation, store assertions, snapshot testing against a headless notcurses instance) so you can test widget behaviour without a terminal.
Richard Hainsworth says
I am really pleased to let you know that finally Damian Conway and I have significantly upgraded the specification of RakuDoc V2, so much so that I am calling it informally v2+. Elizabeth Mattijsen has added extra directives into RakuAST to accommodate these upgrades. I have upgraded Rakuast::RakuDoc::Render so that it now handles the whole RakuDoc V2 specification, and bumped the module version to v1.0.0.
There is a docker image (https://docker.io/finanalyst/browser-editor) that will serve localhost html to a browser and can be used to edit and evaluate RakuDoc. There is also a docker image (https://docker.io/finanalyst/rakudoc_browser) that will do the same for a web-based version.
For members of the Raku community, I have the web based version running at https://raku.finanalyst.org/rakudoc_editor/ This URL can be shared with the Raku community in the weekly, but not more widely.
I encourage anyone who is interested in Rakudoc to test drive the great new enumeration features at the links above. Awesome work by Richard, Damian and Elizabeth.
Don’t Miss the Perl and Raku Conference 2026 in Greenville, SC
SAVE THE DATES! Friday through Sunday, June 26-28
Registration is open: https://tprc.us/tprc-2026-gsp
Weekly Challenge #370 is available for your joy.
This week, Anton Oks has proposed that we focus on some cool Raku from over 200 examples featured on https://rosettacode.org/wiki/Category:Raku. This is his contribution to get the ball rolling (disclosure he is the author of this entry)…
#| Recursive, parallel, random pivot, single-pass, quicksort implementation
multi quicksort(\a where a.elems < 2) { a }
multi quicksort(\a, \pivot = a.pick) {
my %prt{Order} is default([]) = a.classify: * cmp pivot;
my Promise $less = start { samewith(%prt{Less}) }
my $more = samewith(%prt{More});
await $less andthen |$less.result, |%prt{Same}, |$more;
}
This code shows off a cool set of Raku features:
#|.pick)\aa.elems or a.pickis default() in Hash creation, here using an empty array []%prt{Order} or my Promise $less.classify for splitting array using mighty cmpPromise and awaitsamewith() – factoring out the actual function nameandthenVery nice – thanks to Anton for creating some aspirational[*] Raku.
Your contribution is welcome, please make a gist and share via the #raku channel on IRC or Discord.
.WHAT and (.WHAT) both as (Any) by psychoslave[*] Aspirational. Just to clarify, it is not that we should all aspire to that style, but that if this is a style you want – a taut, compact routine that engages all facets of your language, then Raku excels. Other coding styles are available in Raku.
Anton shared these words with me:
… I believe no other language can do that so short and still readable.
Please keep staying safe and healthy, and keep up the good work! Even after week 64 of hopefully only 209.
~librasteve
P.S. Here is why I think Raku is so much fun…
Asking “Why wasn’t Shakespeare a German?” can be reframed as a question about language: why did his genius emerge in English rather than in German? One answer lies in the remarkable flexibility and absorptive quality of English during the late Renaissance. By Shakespeare’s time, English had already drawn heavily from Latin, French, and other languages, creating a hybrid vocabulary that allowed for nuance, invention, and wordplay. Shakespeare could shift easily between high and low registers, coin new expressions, and layer meanings in ways that felt natural within this fluid linguistic system.
German, by contrast—though rich and expressive—was less standardized in the 16th century and often more structurally rigid. Its grammatical complexity and regional fragmentation made it harder to achieve the same kind of rapid, playful experimentation that Shakespeare employed. Where English encouraged improvisation and borrowing, German tended to preserve clearer boundaries within its forms, limiting linguistic elasticity. Shakespeare’s writing thrives on ambiguity, puns, and tonal shifts, and these qualities depended on a language as adaptable as English.
Post image attribution: Eddie Leslie from Lancashire, CC BY-SA 2.0 https://creativecommons.org/licenses/by-sa/2.0, via Wikimedia Commons
This week sees the release of 3 (yes, 3) new Whateverables.
The Whateverables are a collection of IRC bots primarily useful for Raku developers. They are written in Raku and are based on IRC::Client. Many of the use cases involve running Raku code using pre-built versions of the Rakudo compiler for each commit.
They work over the IRC-Discord bridge too mostly (make sure you are in the main #raku channel and note that sometimes the gremlins can get under the bridge – or is that the trolls).
Oh – and have something for the weekly? Then the Notable bot can be fed using weekly: my hot news
Don’t Miss the Perl and Raku Conference 2026 in Greenville, SC
SAVE THE DATES! Friday through Sunday, June 26-28
Registration is open: https://tprc.us/tprc-2026-gsp
Weekly Challenge #369 is available for your giggles.
This week, my eye was caught by an interesting post on Quora by Jan M Savage: Why do we use variables in programming languages?
Variables capture data that we can re-use, but more importantly, judicious use of variables lets our code tell a story.
sub sec-parser($secs) {
my $rsecs = $secs mod 60;
my $rmins = ($secs div 60) mod 60;
my $rhrs = (($secs div 60) div 60) mod 24;
my $days = (($secs div 60) div 60) div 24;
(:$days, :$rhrs, :$rmins, :$rsecs);
}
say sec-parser(240_000); # (days => 2 hrs => 18 mins => 40 secs => 0)
Of course, you’ve made use of variables (declared with my) in order to capture data, but: your code is not telling a story since the variables are poorly named; pray what is rhrs???
Jan will show you how to write the same function such that all the div and mod operations occur underground (so to speak), so that all you can see is what matters:
sub sec-parser($total-sec) {
my ($secs, $mins, $hrs, $days) = $total-sec.polymod(60, 60, 24);
(:$days, :$hrs, :$mins, :$secs);
}
say sec-parser(240_000); #(days => 2 hrs => 18 mins => 40 secs => 0)
This code shows off a cool set of Raku features:
my declaration.polymod method to apply the mod operation multiple times:$hrs to output the name and value of each itemVery nice – thanks to Jan for combining great advice on variables and showing off some Raku.
Your contribution is welcome, please make a gist and share via the #raku channel on IRC or Discord.
As we say in London, it’s like you wait forever for a bus and then 3 come along all at once. So it is with the spasm of activity this week on Whateverables (the last updates to that wiki were in 2023 … and 2020). Thanks to the authors for reviving the fun.
Please keep staying safe and healthy, and keep up the good work! Even after week 63 of hopefully only 209.
~librasteve
Habere-et-Dispertire has shared some new notes on using the raku command:
raku -ne "put .trim" {{/path/to/file}}
raku -ne '.put if "START" ff "STOP"' {{/path/to/file}}
raku -ne '.put if "START" ^ff^ "STOP"' {{/path/to/file}}
Don’t Miss the Perl and Raku Conference 2026 in Greenville, SC
SAVE THE DATES! Friday through Sunday, June 26-28
Registration is open: https://tprc.us/tprc-2026-gsp
Weekly Challenge #368 is available for your entertainment.
This week, my eye was caught by what looked like a simple question on the IRC (Discord) channels:
my (%a, @b, %c, @d) = (1, 2).map({ |({"a" => 1}, [1]) });
Why does this seeming straightforward list assignment with = (see docs) cause an error?
Odd number of elements found where hash initializer expected:Found 5 (implicit) elements: ...
Let’s tease that apart, say we make the LHS just a single Array @b and then a single Hash %a
my @b = (1, 2).map({ |({"a" => 1}, [1]) });
#OUTPUT say @b; #[{a => 1} [1] {a => 1} [1]]
my %a = (1, 2).map({ |({"a" => 1}, [1]) });
#ERROR
The docs say that list assignment is governed by the LHS – an Array will take all the RHS elements. We can surmise that a Hash will take all the RHS elements – even splitting key, values – and translate them into key => value pairs. Thus the error if presented with an odd number in total.
Despite this frustration, Raku has two ways to make this work:
my (%a, @b, %c, @d) Z= (1, 2).map({ |({"a" => 1}, [1]) });
#-or-
my (%a, @b, %c, @d) := (1, 2).map({ |({"a" => 1}, [1]) });
say {:%a,:@b,:%c,:@d}
#OUTPUT {a => {a => 1}, b => [1], c => {a => 1}, d => [1]}
You can use binding := instead and since the LHS and the RHS have the same data structure this will work. Or you can combine Z (the zip metaoperator) with = (item assignment), like this:
my (a, b, c) Z= (a, b, c);
#turns that assignment into boring
my a = @x[0]; my b = @x[1]; my c = @x[2]
Your contribution is welcome, please make a gist and share via the #raku channel on IRC or Discord.
say 𐄊+ ᮳; by hwayneA quiet-ish week this week. Kudos to Anton for his productive spurt on the module front. Happy Easter to those who celebrate.
Please keep staying safe and healthy, and keep up the good work! Even after week 62 of hopefully only 209.
~librasteve
On behalf of the Rakudo development team, I’m happy to announce the March 2026 release of Rakudo #191. Rakudo is an implementation of the Raku language.
The source tarball for this release is available from https://rakudo.org/files/rakudo. Pre-compiled archives will be available shortly.
Improvements:
Fixes:
Additions:
Internal:
The following people contributed to this release:
Elizabeth Mattijsen, Will Coleda, Daniel Green, Justin DeVuyst, David Schultz, Eric Forste, Márton Polgár, Patrick Böker, Richard Hainsworth, ab5tract
Don’t Miss the Perl and Raku Conference 2026 in Greenville, SC
SAVE THE DATES! Friday through Sunday, June 26-28
Registration is open: https://tprc.us/tprc-2026-gsp
The Raku Programming Language Collect, Conserve and Remaster Project – lizmat has remastered…
On Troll Hugging, Hole Digging, and Improving Open Source Communities
Support for fuzzing moar itself with AFL
Dipped my toes in #AFLplusplus over the weekend was a kinda nice distraction / pastime.
Got #moarvm (language runtime for #rakulang) its first fuzzing target that exercises the streaming decoder system which is what you would use when reading strings (rather than bytes) with some encoding from a streaming source like the output of a process you’re running.
If you are in a generous mood and would like to see WWW::Anthropic as a Raku module, then please take note:
BTW, if someone is willing to “donate” Anthropoc API key to me — for a day or two — I can implement “WWW::Anthropic”
Anton is on the Discord / IRC bridge for privmsg purposes.
CragCLI – Command Line Calculator – Examples
I recently shared my fun project here (https://www.reddit.com/r/CLI/). Someone commented “OK it looks cool, but what can I do with it”, so now sharing an example (4 gifs) to demonstrate the potential.
Weekly Challenge #367 is available for your entertainment.
This week, Will Coleda amuses and informs us with their App::Unicode::Mangle module. Despite the obvious -Ofun applications, this is a great inspiration for folks who are interested in how to apply unicode props.
$ - -- '#rakulang'#ⓡⓐⓚⓤⓛⓐⓝⓖ$ - -- 'Hello, github!'¡ɥʇıƃ ,ʃʃǝ$ - -- 'A bird, a plane.'𝐀 𝐛𝐢𝐫𝐝, 𝐚 𝐩𝐥𝐚𝐧𝐞.$ - -- 'lisplike'⒧⒤⒮⒫⒧⒤⒦⒠$ - -- 'combo breaker'̸̩̫̍ͧͮ̄͋͘͠ ̣͚͠ͅř̗ẻ͔̥͎̦ͪ̀̒͋͢$ - -- 'The Telltale Heart'𝘛𝘩𝘦 𝘛𝘦𝘭𝘭𝘵𝘢𝘭𝘦 𝘏𝘦𝘢𝘳𝘵$ - -- 'Presenting'🄿 🅁 🄴 🅂 🄴 🄽 🅃 🄸 🄽 🄶$ - -- 'A little boxy'🅰 🅻 🅸 🆃 🆃 🅻 🅴 🅱 🅾 ❎ 🆈$ - -- 'Happy Birthday!'Ⓗ⒜ⓟ𝐩𝐲 𐐒⒤𝐫⒯⒣ɐ⒴¡
Your contribution is welcome, please make a gist and share via the #raku channel on IRC or Discord.
Congrats to the core team for maintaining their steady release tempo. Great to see the RakuAST mineshaft is getting a little deeper each time.
Zoffix’s post on OSS communities was a foundational read for me back in the day. Thank’s to lizmat and her CCR project for remastering this to a usable state. Please do reach out and help on this if you can.
Please keep staying safe and healthy, and keep up the good work! Even after week 61 of hopefully only 209.
~librasteve
Breaking news: TPRC Talk Submission Deadline extended to April 21, 2026. Many thanks to the organisers for heeding our call for more time.
We are re-opening the talk submissions for TPRC with a new deadline of April 21, 2026. Please submit your 20 minute talks, and 50 minute talks at https://tprc.us/. Let us know if you need help with your submission or your talk development, because we have mentors who can listen to your ideas and guide you.
We are also taking submissions for interactive sessions. These are sessions that have a theme, but invite maximum audience participation; sessions which take advantage of the gathering of community members that have a wide range of experience and ideas to share. You would introduce the theme and moderate the session. If you have ideas for interactive sessions, but don’t want to moderate them yourself, please go to our wiki to enter your ideas, and maybe someone else will pick up the ball!
Don’t Miss the Perl and Raku Conference 2026 in Greenville, SC
SAVE THE DATES! Friday through Sunday, June 26-28
Registration is open: https://tprc.us/tprc-2026-gsp
28th German Perl/Raku Workshop (16th-18th March 2026 in Berlin) took place last week.
Here are some photos from the pre-conference social:
I look forward to seeing the videos and presentations, meantime here is the list to whet your appetite.
Raku community(?) in 2026 – how my permanent ban came to be – Marton Polgar
https://programming.dev/post/47337716
This presentation shows how using Raku’s interactivity and inter-op packages scientific plots and artistic images can be created and generated.
rakulang goes post-quantum! I added a toy implementation of the ML-DSA digital signature algorithm to the Obscure #cryptography library https://github.com/gdncc/Obscure
Obscure: a Raku Programming Language Cryptography Playground
Weekly Challenge #366 is available for your amusement.
One of the hidden gems of the Raku docs is the generated typegraph. Here is an example, the Allomorph type graph:
You can find these at the end of the docs page for most classes (blue – does Role, black – is Class).
Let’s say you have gone to the trouble of writing something like this, and you are still relatively new to raku:
sub ( $x) { $x * 2 } 2; #4 '2'; #Calling dub(Str) will never work with declared signature (Int $x) <2>; #4 <2>; #IntStr.new(2, "2")
A skim read of the docs, suggests that <> angle brackets are for word quoting, and it is easy to get the idea that they would make <2> into '2'. And so it is, at first, surprising that dub <2> succeeds in passing an Int type check.
Actually since you have all now read Lizmats great advent post on Q constructs, you will know that <> calls the val() function on each word. And that makes an Allomorph if the word can be construed as a string with a value. In this case, an IntStr.
Allomorph classes should be considered a common raku scenario. If you are thinking “I checked for Int and I got something that is not an Int (but inherits from Int)”, then this is a general misunderstanding about how type checking and class inheritance works and is not limited to Allomorphs.
Consider this code:
{} {} {}sub ( $a) { "{$a.^name}s can't read" }my $h = .new;say $h; #Humans can't read
No Allomorphs in sight, but it does the “surprising” thing of letting a Human pass the Animal type check.
Your contribution is welcome, please make a gist and share via the #raku channel on IRC or Discord.
Many thanks to our new sponsors – visit https://raku.org and follow the links to check them out.
Please keep staying safe and healthy, and keep up the good work! Even after week 60 of hopefully only 209.
~librasteve
use Math::NumberTheory;use BigRoot;use Image::Markup::Utilities;use Graphviz::DOT::Chessboard;use Data::Reshapers;use JavaScript::D3;use JavaScript::D3::Utilities;
#%javascriptrequire.config({ paths: { d3: 'https://d3js.org/d3.v7.min'}});require(['d3'], function(d3) { console.log(d3);});
my $title-color = 'Ivory';my $stroke-color = 'SlateGray';
The built-in Raku constant pi (or π) is fairly low precision:
say π.fmt('%.25f')
# 3.1415926535897930000000000
One way to remedy that is to use continued fractions. For example, using the (first) sequence line of On-line Encyclopedia of Integer Sequences (OEIS) A001203 produces with precision 56:
my @s = 3, 7, 15, 1, 292, 1, 1, 1, 2, 1, 3, 1, 14, 2, 1, 1, 2, 2, 2, 2, 1, 84, 2, 1, 1, 15, 3, 13, 1, 4, 2, 6, 6, 99, 1, 2, 2, 6, 3, 5, 1, 1, 6, 8, 1, 7, 1, 2, 3, 7, 1, 2, 1, 1, 12, 1, 1, 1, 3, 1, 1, 8, 1, 1, 2, 1, 6, 1, 1, 5, 2, 2, 3, 1, 2, 4, 4, 16, 1, 161, 45, 1, 22, 1, 2, 2, 1, 4, 1, 2, 24, 1, 2, 1, 3, 1, 2, 1;my $pi56 = from-continued-fraction(@s».FatRat.List);
# 3.14159265358979323846264338327950288419716939937510582097
Here we verify the precision using Wolfram Language:
"wolframscript -code 'N[Pi, 100] - $pi56'"andthen .&shell(:out)andthen .out.slurp(:close)
# 0``56.
More details can be found in Wolfram MathWorld page “Pi Continued Fraction”, [EW1].
It is interesting to consider the plotting the terms of continued fraction terms of .
First we ingest the more “pi-terms” from OEIS A001203 (20k terms):
my @ds = data-import('https://oeis.org/A001203/b001203.txt').split(/\s/)».Int.rotor(2);my @terms = @ds».tail;@terms.elems
# 20000
Here is the summary:
sink records-summary(@terms)
# +-------------------+# | numerical |# +-------------------+# | 1st-Qu => 1 |# | Median => 2 |# | Min => 1 |# | Max => 20776 |# | Mean => 12.6809 |# | 3rd-Qu => 5 |# +-------------------+
Here is an array plot of the first 128 terms of the continued fraction approximating :
#% htmlmy @mat = |@terms.head(128)».&integer-digits(:2base);my $max-digits = @mat».elems.max;@mat .= map({ [|(0 xx (``max-digits - ``_.elems)), |$_] });dot-matrix-plot(transpose(@mat), size => 10):svg
Next, we show the Pareto principle manifestation of for the continued fraction terms. First we observe that the terms a distribution similar to Benford’s law:
#% jsmy @tally-pi = tally(@terms).sort(-*.value).head(16) <</>> @terms.elems;my @terms-b = random-variate(BenfordDistribution.new(:10base), 2_000);my @tally-b = tally(@terms-b).sort(-*.value).head(16) <</>> @terms-b.elems;js-d3-bar-chart( [ |@tally-pi.map({ %( x => ``_.key, y => ``_.value, group => 'π') }), |@tally-b.map({ %( x => ``_.key, y => ``_.value, group => 'Benford') }) ], plot-label => "Pi continued fraction terms vs. Benford's law", :$title-color, :$background)
Here is the Pareto principle plot — ≈5% of the unique term values correspond to ≈80% of the terms:
#% jsjs-d3-list-line-plot( pareto-principle-statistic(@terms), plot-label => "Pi continued fraction terms vs. Benford's law", :$title-color, :$background, stroke-width => 5, :grid-lines)
Many ways to express π as an infinite sum — some converge slowly, others surprisingly fast.
Leibniz–Gregory series (1671/ Madhava earlier)
Raku implementation:
sub pi-leibniz($n) { 4 * [+] map { (``_ %% 2 ?? 1 !! -1) / (2 * ``_.FatRat + 1) }, 0 ..^ $n}my $piLeibniz = pi-leibniz(1_000);
# 3.140592653839792925963596502869395970451389330779724489367457783541907931239747608265172332007670207231403885276038710899938066629552214564551237742887150050440512339302537072825852760246628025562008569471700451065826106184744099667808080815231833582150382088582680381403109153574884416966097481526954707518119416184546424446286573712097944309435229550466609113881892172898692240992052089578302460852737674933105951137782047028552762288434104643076549100475536363928011329215789260496788581009721784276311248084584199773204673225752150684898958557383759585526225507807731149851003571219339536433193219280858501643712664329591936448794359666472018649604860641722241707730107406546936464362178479780167090703126423645364670050100083168338273868059379722964105943903324595829044270168232219388683725629678859726914882606728649659763620568632099776069203461323565260334137877
Verify with Wolfram Language (again):
"wolframscript -code 'N[Pi, 1000] - $piLeibniz'"andthen .&shell(:out)andthen .out.slurp(:close)
# 0.000999999750000312499...814206`866.9999998914263
Nilakantha series (faster convergence):
Raku:
sub pi-nilakantha($n) { 3 + [+] map { ($_ %% 2 ?? -1 !! 1 ) * 4 / ((2 * $_.FatRat) * (2 * $_ + 1) * (2 * $_ + 2)) }, 1 .. $n}pi-nilakantha(1_000);
# 3.141592653340542051900128736253203567152539255317954874674304859504426172618558702218695071137605738966036069683335561974900086119307836254205910905806190030949758215864755464129701335459521079534522811851010296642538249613529207613335816447914992502190861349451746347920350033634355181084537761886275546599078437173552420948534950023442771396391252038722980428723971632669306434394851189528826699233048019261441283970866004550291393472342649870962106821115715774722114776992400455398838055772839725805047379519366309217982783671029012753365224924699602163737619311405432798527164991008945233085366633073462699045511265528492985424805854418596455931463431855615794431867539190155631617285217459790661344075940516099637034367441911754544671168909454186231972510120715400925996293656987342326715209388299050131213232932065481743222390684073879385764855135985734675127240826
"wolframscript -code 'N[Pi, 1000] - {pi-nilakantha(1_000)}'"andthen .&shell(:out)andthen .out.slurp(:close)
# 2.4925118...83814206`860.3966372344514*^-10
Wallis product (1655) — elegant infinite product:
Raku running product:
my $p = 2.0;for 1 .. 1_000 -> $n { ``p *= (2 * ``n) * (2 * ``n) / ( (2 * ``n - 1 ) * ( 2 * $n + 1) ); say "``n → {``p / ``piLeibniz} relative error" if ``n %% 100;}
# 100 → 0.9978331595460779 relative error# 200 → 0.9990719099195204 relative error# 300 → 0.9994865459690567 relative error# 400 → 0.9996941876848563 relative error# 500 → 0.9998188764663584 relative error# 600 → 0.9999020455903246 relative error# 700 → 0.9999614733132168 relative error# 800 → 1.0000060557070767 relative error# 900 → 1.0000407377794782 relative error# 1000 → 1.000068487771041 relative error
One of the fastest-converging series used in record computations:
Each term adds roughly 14 correct digits. Cannot be implemented easily in Raku, since Raku does not have bignum sqrt and power operations.
Spigot algorithms compute decimal digits using only integer arithmetic — no floating-point errors accumulate.
The classic Rabinowitz–Wagon spigot (based on a transformed Wallis product) produces base-10 digits sequentially.
Simple (but bounded) version outline in Raku:
sub spigot-pi($digits) { my ``len = (10 * ``digits / 3).floor + 1; my @a = 2 xx $len; my @result; for 1..$digits { my $carry = 0; for ``len-1 ... 0 -> ``i { my ``x = 10 * @a[``i] + ``carry * (``i + 1); @a[``i] = ``x % (2 * $i + 1); ``carry = ``x div (2 * $i + 1); } @result.push($carry div 10); @a[0] = $carry % 10; # (handle carry-over / nines adjustment in full impl) } @result.head(1).join('.') ~ @result[1..*].join}spigot-pi(50);
# 314159265358979323846264338327941028841971693993751
"wolframscript -code 'N[Pi, 100] - {spigot-pi(50).FatRat / 10e49.FatRat}'"andthen .&shell(:out)andthen .out.slurp(:close)
# 2.3969628881355243801510070603398913366797194459230781640628621`41.37966130996076*^-16
Bailey–Borwein–Plouffe (1995) formula lets you compute the nth hexadecimal digit of π directly (without earlier digits):
Very popular for distributed π-hunting projects. The best known digit-extraction algorithm.
Raku snippet for partial sum (base 16 sense):
sub bbp-digit-sum($n) { [+] (0..$n).map: -> $k { my $r = 1/16**$k; $r * (4/(8*$k+1) - 2/(8*$k+4) - 1/(8*$k+5) - 1/(8*$k+6)) }}say bbp-digit-sum(100).base(16).substr(0,20);
# 3.243F6B
Let us plot a random walk using the terms of continued fraction of Pi — the 20k or OEIS A001203 — to determine directions:
#% jsmy @path = angle-path(@terms)».reverse».List;my &pi-path-map = { given @terms[$_] // 0 { when $_ ≤ 100 { 0 } when $_ ≤ 1_000 { 1 } default { 2 } } }@path = @path.kv.map( -> $i, $p {[|$p, &pi-path-map($i).Str]});my %opts = color-scheme => 'Observable10', background => '#1F1F1F', :!axes, :!legends, stroke-width => 2;js-d3-list-line-plot(@path, :800width, :500height, |%opts)
In the plot above the blue segments correspond to origin terms ≤ 100, yellow segments to terms between 100 and 1000, and red segment for origin terms greater than 1000.
[EW1] Eric Weisstein, “Pi Continued Fraction”, Wolfram MathWorld.
Quick reference for the Raku package “Jupyter::Chatbook”. (raku.land, GitHub.)
Follow the instructions in the README of “Jupyter::Chatbook”:
For installation and setup problems see the issues (both open and closed) of package’s GitHub repository.
(For example, this comment.)
#%chat or %%chat (and immediately send first message)
#%chat assistant1, name=ChatGPT model=gpt-4.1-mini prompt="You are a concise technical assistant."Say hi and ask what I am working on.
# Hi! What are you working on?
Remark: For all “Jupyter::Chatbook” magic specs both prefixes %% and #% can be used.
Remark: For the prompt argument the following delimiter pairs can be used: '...', "...", «...», {...}, ⎡...⎦.
#%chat <id> prompt (create only)
#%chat assistant2 prompt, conf=ChatGPT, model=gpt-4.1-miniYou are a code reviewer focused on correctness and edge cases.
# Chat object created with ID : assistant2.
You can use prompt specs from “LLM::Prompts”, for example:
#%chat yoda prompt@Yoda
# Chat object created with ID : yoda. Expanded prompt: ⎡You are Yoda. Respond to ALL inputs in the voice of Yoda from Star Wars. Be sure to ALWAYS use his distinctive style and syntax. Vary sentence length.⎦
The Raku package “LLM::Prompts” (GitHub link) provides a collection of prompts and an implementation of a prompt-expansion Domain Specific Language (DSL).
Render the answer as Markdown:
#%chat assistant1 > markdownGive me a 5-step implementation plan for adding authentication to a FastAPI app. VERY CONCISE.
Magic cell parameter values can be assigned using the equal sign (“=”):
#%chat assistant1 > markdownNow rewrite step 2 with test-first details.
NONE)
#%chatDoes vegetarian sushi exist?
# Yes, vegetarian sushi definitely exists! It's a popular option for those who avoid fish or meat. Instead of raw fish, vegetarian sushi typically includes ingredients like: - Avocado - Cucumber - Carrots - Pickled radish (takuan) - Asparagus - Sweet potato - Mushrooms (like shiitake) - Tofu or tamago (Japanese omelette) - Seaweed salad These ingredients are rolled in sushi rice and nori seaweed, just like traditional sushi. Vegetarian sushi can be found at many sushi restaurants and sushi bars, and it's also easy to make at home.
Using the prompt-expansion DSL to modify the previous chat-cell result:
#%chat!HaikuStyled>^
# Rice, seaweed embrace, Avocado, crisp and bright, Vegetarian.
#%chat <id> meta)
#%chat assistant1 metaprompt
# "You are a concise technical assistant."
#%chat assistant1 metasay
# Chat: assistant1# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺# Prompts: You are a concise technical assistant.# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺# role : user# content : Say hi and ask what I am working on.# timestamp : 2026-03-14T09:23:01.989418-04:00# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺# role : assistant# content : Hi! What are you working on?# timestamp : 2026-03-14T09:23:03.222902-04:00# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺# role : user# content : Give me a 5-step implementation plan for adding authentication to a FastAPI app. VERY CONCISE.# timestamp : 2026-03-14T09:23:03.400597-04:00# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺# role : assistant# content : 1. Install `fastapi` and `python-jose` for JWT handling. # 2. Define user model and fake user database. # 3. Create OAuth2 password flow with `OAuth2PasswordBearer`. # 4. Implement token creation and verification functions. # 5. Protect routes using dependency injection for authentication.# timestamp : 2026-03-14T09:23:05.106661-04:00# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺# role : user# content : Now rewrite step 2 with test-first details.# timestamp : 2026-03-14T09:23:05.158446-04:00# ⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺⸺# role : assistant# content : 2. Write tests to verify user data retrieval and password verification; then define user model and fake user database accordingly.# timestamp : 2026-03-14T09:23:06.901396-04:00
# Bool::True
#%chat allkeys
# NONE assistant1 assistant2 ce gc html latex raku yoda
#%chat allgist
# {NONE => LLM::Functions::Chat(chat-id = NONE, llm-evaluator.conf.name = chatgpt, messages.elems = 4, last.message = ${:content("Rice, seaweed embrace, \nAvocado, crisp and bright, \nVegetarian."), :role("assistant"), :timestamp(DateTime.new(2026,3,14,9,23,10.770353078842163,:timezone(-14400)))}), assistant1 => LLM::Functions::Chat(chat-id = assistant1, llm-evaluator.conf.name = ChatGPT, messages.elems = 6, last.message = ${:content("2. Write tests to verify user data retrieval and password verification; then define user model and fake user database accordingly."), :role("assistant"), :timestamp(DateTime.new(2026,3,14,9,23,6.901396036148071,:timezone(-14400)))}), assistant2 => LLM::Functions::Chat(chat-id = assistant2, llm-evaluator.conf.name = chatgpt, messages.elems = 0), ce => LLM::Functions::Chat(chat-id = ce, llm-evaluator.conf.name = chatgpt, messages.elems = 0), gc => LLM::Functions::Chat(chat-id = gc, llm-evaluator.conf.name = chatgpt, messages.elems = 0), html => LLM::Functions::Chat(chat-id = html, llm-evaluator.conf.name = chatgpt, messages.elems = 0), latex => LLM::Functions::Chat(chat-id = latex, llm-evaluator.conf.name = chatgpt, messages.elems = 0), raku => LLM::Functions::Chat(chat-id = raku, llm-evaluator.conf.name = chatgpt, messages.elems = 0), yoda => LLM::Functions::Chat(chat-id = yoda, llm-evaluator.conf.name = chatgpt, messages.elems = 0)}
#%chat assistant1 metadelete
# Deleted: assistant1 Gist: LLM::Functions::Chat(chat-id = assistant1, llm-evaluator.conf.name = ChatGPT, messages.elems = 6, last.message = ${:content("2. Write tests to verify user data retrieval and password verification; then define user model and fake user database accordingly."), :role("assistant"), :timestamp(DateTime.new(2026,3,14,9,23,6.901396036148071,:timezone(-14400)))})
#%chat assistant2 metaclear
# Cleared messages of: assistant2 Gist: LLM::Functions::Chat(chat-id = assistant2, llm-evaluator.conf.name = chatgpt, messages.elems = 0)
#%chat alldrop
# Deleted 8 chat objects with names NONE assistant2 ce gc html latex raku yoda.
#%chat <id>|all meta command aliases / synonyms:
delete or dropkeys or namesclear or empty#%chat)prompt.conf (default: ChatGPT).#%openai, %%gemini, %%llama, %%dalle)#%chat.Remark: For all “Jupyter::Chatbook” magic specs both prefixes %% and #% can be used.
#%openai > markdown, model=gpt-4.1-miniWrite a regex for US ZIP+4.
#%gemini > markdown, model=gemini-2.5-flashExplain async/await in Python using three point each with less than 10 words.
Access llamafile, locally run models:
#%llama > markdown Give me three Linux troubleshooting tips. VERY CONCISE.
Remark: In order to run the magic cell above you have to run a llamafile program/model on your computer. (For example, ./google_gemma-3-12b-it-Q4_K_M.llamafile.)
Access Ollama models:
#%chat ollama > markdown, conf=OllamaGive me three Linux troubleshooting tips. VERY CONCISE.
Remark: In order to run the magic cell above you have to run an Ollama app on your computer.
Create images using DALL-E:
#%dalle, model=dall-e-3, size=landscapeA dark-mode digital painting of a lighthouse in stormy weather.
For a detailed discussion of the DALL-E interaction in Raku and magic cell parameter descriptions see “Day 21 – Using DALL-E models in Raku”.
Image generation:
#%dalle, model=dall-e-3, size=landscape, style=vividA dark-mode digital painting of a lighthouse in stormy weather.
Here we use a DALL-E meta cell to see how many images were generated in a notebook session:
#% dalle metaelems
# 3
Here we export the second image — using the index 1 — into a file named “stormy-weather-lighthouse-2.png”:
#% dalle export, index=1stormy-weather-lighthouse-2.png
# stormy-weather-lighthouse-2.png
Here we show all generated images:
#% dalle metashow
Here we export all images (into file names with the prefix “cheatsheet”):
#% dalle export, index=all, prefix=cheatsheet
API keys can be passed inline (api-key) or through environment variables.
%*ENV<OPENAI_API_KEY> = "YOUR_OPENAI_KEY";%*ENV<GEMINI_API_KEY> = "YOUR_GEMINI_KEY";%*ENV<OLLAMA_API_KEY> = "YOUR_OLLAMA_KEY";
Ollama-specific defaults:
OLLAMA_HOST (default host fallback is http://localhost:11434)OLLAMA_MODEL (default model if model=... not given)The magic cells take as argument base-url. This allows to use LLMs that have ChatGPT compatible APIs. The argument base_url is a synonym of host for magic cell #%ollama.
Initialization runs when the extension is loaded.
RAKU_CHATBOOK_INIT_FILE~/.config/raku-chatbook/init.py~/.config/init.rakuUse this for imports/helpers you always want in chatbook sessions.
RAKU_CHATBOOK_LLM_PERSONAS_CONF~/.config/raku-chatbook/llm-personas.json~/.config/llm-personas.jsonThe supported JSON shape is an array of dictionaries:
[ { "chat-id": "raku", "conf": "ChatGPT", "prompt": "@CodeWriterX|Raku", "model": "gpt-4.1-mini", "max_tokens": 8192, "temperature": 0.4 }]
Recognized persona spec fields include:
chat-idpromptconf (or configuration)model, max-tokens, temperature, base-urlapi-keyevaluator-args (object)Verify pre-loaded personas:
#%chat allkeys
A Calculator for the Command Line, that grabs data directly from AI.
You are a command line user, perhaps a software developer, a data scientist or an IT savvy engineer.
You are a numerate person, who likes to know the numbers behind what you see.
You often reach for a calculator. You are comfortable driving a scientific calculator. Maybe you already use another command line calculator like bc.
CragCLI is right there.
> crag '0.1+0.2' #output 0.3
It does two things.
Text based scientific calculator….
Arithmetic, Functions, Logs, Trig, Complex, Random, Memory, Types, Rounding
…with next level features, such as artificial intelligence and dimensional analysis[1].
LLM, SIUnits, NonSI, Time, Data, Currency, Angles, Conversions, Errors, Constants, Colors, Sequences, Bases, Bitwise, Unicode, Theory
CragCLI also provides an interactive mode via a REPL[2].
Visit the CragCLI website to see the interactive mode in action and follow the Getting Started instructions.
By the time I read Raku: A Language for Gremlins, the idea for Crag had been simmering away. Here’s what Hillel Wayne has to say:
I finally checked it out last week to see if it’d make a good “calculator language”. I use a hodgepodge of Python, J, Frink, and Excel to do math and they all have their own big drawbacks, so it’d be nice if Raku could round them out.
After several days of experiments, I’m at a loss of how to describe Raku. The best I can come up with is that the language was designed by a bunch of really intelligent gremlins. Gremlins who spent a lot of time gathering feedback from other gremlins.
That lead me to Frink. It was a pleasant surprise for me to learn that there are other Gremlins out there, not just me.
My way in to Raku was to implement a couple of modules – plenty of OO and Grammar learnings and a reflection of my physics mindset.
These were already done when Hillel piped up.
This Raku package provides functions and function objects to access, interact, and utilize Large Language Models (LLMs), likeOpenAI, Gemini, MistralAI, and Ollama.
Since CragCLI uses many Raku modules, credit is due to all the individual authors and contributors of those. More details in the App::Crag repository.
While there are too many contributors to these tools to list them all, I would like to highlight:
All feedback is welcome, in the comments below or make an Issue on the App::Crag repo.
All done in a spirit of -Ofun.
~librasteve
Notes:
[1] Dimensional Analysis is a method of checking or deriving relationships between physical quantities by comparing their fundamental dimensions (such as mass, length, and time).
[2] REPL is an interactive programming environment that repeatedly Reads user input, Evaluates it, Prints the result, and Loops to wait for the next command.
In this blog post (notebook) we calibrate the Heterogeneous Salvo Combat Model (HSCM), [MJ1, AAp1, AAp2], to the First World War Battle of Coronel, [Wk1]. Our goal is to exemplify the usage of the functionalities of the package “Math::SalvoCombatModeling”, [AAp1]. We closely follow the Section B of Chapter III of [MJ1]. The calibration data used in [MJ1] is taken from [TB1].
Remark: The implementation of the Raku package “Math::SalvoCombatModeling”, [AAp1], closely follows the implementation of the Wolfram Language (WL) paclet “SalvoCombatModeling”, [AAp2]. Since WL has (i) symbolic builtin computations and (ii) a mature notebook system the salvo models computation, representation, and study with WL is much more convenient.
Here we load the package:
use Math::SalvoCombatModeling;use Graph;
The Battle of Coronel is a First World War naval engagement between three British ships {Good Hope, Monmouth, and Glasgow) and four German ships (Scharnhorst, Gneisenau, Leipzig, and Dresden). The battle happened on 1 November 1914, off the coast of central Chile near the city of Coronel.
The Scharnhorst and Gneisenau are the first ships to open fire at Good Hope and Monmouth; the three British ships soon afterwards return fire. Dresden and Leipzig open fire on Glasgow, driving her out of the engagement. At the end of the battle, both Good Hope and Monmouth are sunk, while Glasgow, Scharnhorst, and Gneisenau were damaged.
| Ship | Duration of fire |
|---|---|
| Good Hope | 0 |
| Monmouth | 0 |
| Glasgow | 15 |
| Scharnhorst | 28 |
| Gneisenau | 28 |
| Leipzig | 2 |
| Dresden | 2 |
The following graph shows which ship shot at which ships and total fire duration (in minutes):
#% htmlmy @edges = { from =>'Scharnhorst', to =>'Good Hope', weight => 28 },{ from =>'Scharnhorst', to =>'Monmouth', weight => 28 },{ from =>'Gneisenau', to =>'Good Hope', weight => 28 },{ from =>'Gneisenau', to =>'Monmouth', weight => 28 },{ from =>'Leipzig', to =>'Glasgow', weight => 2 },{ from =>'Glasgow', to =>'Scharnhorst', weight => 2 },{ from =>'Glasgow', to =>'Gneisenau', weight => 15 },{ from =>'Glasgow', to =>'Leipzig', weight => 15 },{ from =>'Glasgow', to =>'Dresden', weight => 15 },{ from =>'Dresden', to =>'Glasgow', weight => 15 };my $g = Graph.new(@edges):directed;$g.dot( engine => 'neato', vertex-shape => 'ellipse', vertex-width => 0.65, :5size, :8vertex-font-size, :weights, :6edge-font-size, edge-thickness => 0.8, arrow-size => 0.6):svg;
Before going with building the model here is table that provides definitions of the fundamental notions of salvo combat modeling:
#% htmlsalvo-notion-definitions('English')==> to-html(field-names => <notion definition>, align => 'left')
| notion | definition |
|---|---|
| Force | A group of naval ships that operate and fight together. |
| Unit | A unit is an individual ship in a force. |
| Salvo | A salvo is the number of shots fired as a unit of force in a discrete period of time. |
| Combat Potential | Combat Potential is a force’s total stored offensive capability of an element or force measured in number of total shots available. |
| Combat Power | Also called Striking Power, is the maximum offensive capability of an element or force per salvo, measured in the number of hitting shots that would be achieved in the absence of degrading factors. |
| Scouting Effectiveness | Scouting Effectiveness is a dimensionless degradation factor applied to a force’s combat power as a result of imperfect information. It is a number between zero and one that describes the difference between the shots delivered based on perfect knowledge of enemy composition and position and shots based on existing information [Ref. 7]. |
| Training Effectiveness | Training effectiveness is a fraction that indicates the degradation in combat power due the lack of training, motivation, or readiness. |
| Distraction Factor | Also called chaff effectiveness or seduction, is a multiplier that describes the effectiveness of an offensive weapon in the presence of distraction or other soft kill. This multiplier is a fraction, where one indicates no susceptibility/complete effectiveness and zero indicates complete susceptibility/no effectiveness. |
| Offensive Effectiveness | Offensive effectiveness is a composite term made of the product of scouting effectiveness, training effectiveness, distraction, or any other factor which represents the probability of a single salvo hitting its target. Offensive effectiveness transforms a unit’s combat potential parameter into combat power. |
| Defensive Potential | Defensive potential is a force’s total defensive capability measured in units of enemy hits eliminated independent of weapon system or operator accuracy or any other multiplicative factor. |
| Defensive Power | Defensive power is the number of missiles in an enemy salvo that a defending element or force can eliminate. |
| Defender Alertness | Defender alertness is the extent to which a defender fails to take proper defensive actions against enemy fire. This may be the result of any inattentiveness due to improper emission control procedures, readiness, or other similar factors. This multiplier is a fraction, where one indicates complete alertness and zero indicates no alertness. |
| Defensive Effectiveness | Defensive effectiveness is a composite term made of the product of training effectiveness and defender alertness. This term also applies to any value that represents the overall degradation of a force’s defensive power. |
| Staying Power | Staying power is the number of hits that a unit or force can absorb before being placed out of action. |
The British ships are in Good Hope, Monmouth, and Glasgow. They correspond to the indices 1, ,2, and 3 respectively.
["Good Hope", "Monmouth", "Glasgow"] Z=> 1..3
# (Good Hope => 1 Monmouth => 2 Glasgow => 3)
The German ships are Scharnhorst, Gneisenau, Leipzig, and Dresden:
["Scharnhorst", "Gneisenau", "Leipzig", "Dresden"] Z=> 1..4
# (Scharnhorst => 1 Gneisenau => 2 Leipzig => 3 Dresden => 4)
Remark: The Battle of Coronel is modeled with a “typical” salvo model — the ships use “continuous fire.” Hence, there are no interceptors and or, in model terms, defense terms or matrices.
Here is the model (for 3 British ships and 4 German ships):
sink my $m = heterogeneous-salvo-model(['B', 3], ['G', 4]):latex;
Remove the defense matrices (i.e. make them zero):
sink $m<B><defense-matrix> = ((0 xx $m<B><defense-matrix>.head.elems).Array xx $m<B><defense-matrix>.elems).Array;sink $m<G><defense-matrix> = ((0 xx $m<G><defense-matrix>.head.elems).Array xx $m<G><defense-matrix>.elems).Array;
Converting the obtained model data structure to LaTeX we get:
Setting the parameter values as in [MJ1] defining the sub param (to be passed heterogeneous-salvo-model):
multi sub param(Str:D $name, Str:D $a where * eq 'B', Str:D $b where * eq 'G', UInt:D $i, UInt:D $j) { 0 }multi sub param(Str:D $name, Str:D $a where * eq 'G', Str:D $b where * eq 'B', UInt:D $i, UInt:D $j) { given $name { when 'beta' { given ($i, $j) { when (1, 1) { 2.16 } when (1, 2) { 2.16 } when (1, 3) { 2.16 } when (2, 1) { 2.16 } when (2, 2) { 2.16 } when (2, 3) { 2.16 } when (3, 1) { 2.165 } when (3, 2) { 2.165 } when (3, 3) { 2.165 } when (4, 1) { 2.165 } when (4, 2) { 2.165 } when (4, 3) { 2.165 } } } when 'curlyepsilon' { given ($i, $j) { when (1, 1) { 0.028 } when (1, 2) { 0.028 } when (1, 3) { 0.028 } when (2, 1) { 0.028 } when (2, 2) { 0.028 } when (2, 3) { 0.028 } when (3, 1) { 0.012 } when (3, 2) { 0.012 } when (3, 3) { 0.012 } when (4, 1) { 0.012 } when (4, 2) { 0.012 } when (4, 3) { 0.012 } } } when 'capitalpsi' { given ($i, $j) { when (1, 1) { 0.5 } when (1, 2) { 0.5 } when (2, 1) { 0.5 } when (2, 2) { 0.5 } when (3, 1) { 0 } when (3, 2) { 0 } when (4, 1) { 0 } when (4, 2) { 0 } when (1, 3) { 0 } when (2, 3) { 0 } when (3, 3) { 1 } when (4, 3) { 1 } } } }}multi sub param(Str:D $name, Str:D $a where * eq 'G', UInt:D $i) { 1 }multi sub param(Str:D $name, Str:D $a where * eq 'B', UInt:D $i) { given $name { when 'zeta' { given $i { when 1 { 1.605 } when 2 { 1.605 } when 3 { 1.23 } } } }}multi sub param(Str:D $name where $name eq 'units', Str:D $a where * eq 'B', UInt:D $i) { $i }multi sub param(Str:D $name where $name eq 'units', Str:D $a where * eq 'G', UInt:D $i) { $i }
# ¶m
my $m = heterogeneous-salvo-model(['B', 3], ['G', 4], :offensive-effectiveness-terms, :¶m)
# {B => {defense-matrix => [[0 0 0] [0 0 0] [0 0 0]], offense-matrix => [[0.018841 0.018841 0 0] [0.018841 0.018841 0 0] [0 0 0.021122 0.021122]], units => [1 2 3]}, G => {defense-matrix => [[0 0 0 0] [0 0 0 0] [0 0 0 0] [0 0 0 0]], offense-matrix => [[0 0 0] [0 0 0] [0 0 0] [0 0 0]], units => [1 2 3 4]}}
$m<B><offense-matrix>
# [[0.018841 0.018841 0 0] [0.018841 0.018841 0 0] [0 0 0.021122 0.021122]]
my $ΔB = $m<B><offense-matrix>».sum
# (0.037682 0.037682 0.042244)
How many salvos to achieve total damage of Good Hope and Monmouth:
1 / $ΔB.head
# 26.537698
That is close to the 28 min of fire by Scharnhorst and Gneisenau at Good Hope and Monmouth.
Total damage of on Glasgow — Leipzig and Dresden fire for 2 min at Glasgow:
$ΔB.tail * 2
# 0.084488
[MJ1] Michael D. Johns, Steven E. Pilnick, Wayne P. Hughes, “Heterogeneous Salvo Model for the Navy After Next”, (2000), Defense Technical Information Center.
[TB1] Thomas R. Beall, “The Development of a Naval Battle Model and Its Validation Using Historical Data”, (1990), Defense Technical Information Center.
[Wk1] Wikipedia entry, Salvo combat model.
[AAp1] Anton Antonov, Math::SalvoCombatModeling, Raku package, (2026), GitHib/antononcube.
[AAp2] Anton Antonov, SalvoCombatModeling, Wolfram Language paclet, (2024), Wolfram Language Paclet Repository.
This blog post has been removed because of legal reasons by request of the author.
This is part thirteen in the "Cases of UPPER" series of blog posts, describing the Raku syntax elements that are completely in UPPERCASE.
This part will discuss the various introspection methods that you can use on objects in the Raku Programming Language. Note that in some documentation these methods are also referred to as Metamethods.
In Raku everything is an object, or can be thought of as an object. An object is an instantiation of a class (usually made by calling the new method on it). A class is represented by a so-called "type object". Such a type object in turn is an instantation of a so-called meta class. And these meta classes are themselves built out of more primitive representations.
Going this deep would most definitely be out of scope for these blog posts. But yours truly does intend to go there at some point in the future.
The WHAT method returns the type object of the given invocant. Not much else to tell about it really.
say 42.WHAT; # (Int)
say "foo".WHAT; # (Str)
say now.WHAT; # (Instant)
The HOW method returns the meta-object of the class of the given invocant. The HOW (b)acronym stands for "Higher Order Workings". It allows one to introspect the class of the invocant.
say 42.HOW.name(42); # Int
say "foo".HOW.name("foo"); # Str
say now.HOW.name(now); # Instant
Note that the invocant of the HOW method needs to be repeated in the introspection method's call as the first argument. Why? Well, this is really to be possibly compatible with future versions of Raku.
Since one is usually only interested in the introspection aspect of HOW, a shortcut method invocation was created that allows one to directly call the introspection method without needing to repeat oneself: .^:
say 42.^name; # Int
say "foo".^name; # Str
say now.^name; # Instant
Some other common introspection methods are mro (showing the base classes of the class of the value) and methods (which returns the method objects of the methods that can be called on the value):
say 42.^mro; # ((Int) (Cool) (Any) (Mu))
say 42.^methods.sort; # (ACCEPTS Bool Bridge Capture Complex...
Note that these meta-classes are classes themselves, so can have meta-methods on them called as well:
say 42.HOW.^name; # Perl6::Metamodel::ClassHOW
Yeah, there's still some legacy code that will need renaming under the hood!
The WHERE method returns the memory address of the invocant. It is of limited use in the Rakudo implementation as the memory location of an object is not guaranteed to be constant. As such, it is intended for (core) debugging only.
say 42.WHERE; # 2912024602280 (or some other number)
In the previous blog post the Scalar object was described. But the Scalar objects are nearly invisible. How can one obtain a Scalar object from a given variable? And find out its name from that?
The "secret" to that is the VAR method.
my $a;
say $a.VAR.^name; # Scalar
Each Scalar object provides at least these introspection methods: of, name, default and dynamic.
my Int $a is default(42) = 666;
say $a; # 666
say $a.VAR.of; # (Int)
say $a.VAR.name; # $a
say $a.VAR.default; # 42
say $a.VAR.dynamic; # False
The of method returns the constraint that needs to be fulfilled in order to be able to assign to the variable. The name method returns the name of the variable. The default method returns the default value (Any if none is specified).
The dynamic method returns True or False whether the variable is visible for dynamic variable lookups. This usually only returns True for variables with the * twigil.
The WHO method (for "who lives here?) is actually a bit of a misnomer. It should probably have been called OUR because it returns the Stash of the type object of the invocant. And a stash is an object that does the Associative role, and as such can be accessed as if it were a Hash. And the stash of a type object is the same namespace as our inside that package.
So for instance if you would like to know all classes that live in the IO package:
say IO.WHO.keys.sort; # (ArgFiles CatHandle Handle Notification Path Pipe Socket Spec Special)
Of course, you would know them more by their complete names such as IO::ArgFiles, IO::CatHandle, IO::Handle, etc. In fact the :: delimiter is shortcut for using WHO:
say IO.WHO<Handle>; # (Handle)
say IO::Handle; # (Handle)
And that goes even further: foo:: is just short for foo.WHO:
say IO::.keys.sort; # (ArgFiles CatHandle Handle Notification Path Pipe Socket Spec Special)
A little closer to home: how would that look in a package that you define yourself and have an our scoped variable in there:
package A {
our $foo = 42;
}
say A.WHO<$foo>; # 42
say A::<$foo>; # 42
say $A::foo; # 42
The careful reader will have noticed that
packagewas used in the example. A very simple reason:class,role,grammarare all just packages with differentHOWs. AndWHOdoesn't care why kind ofpackageit is.
The REPR method returns the name of the memory representation of the class of the invocant. For most of the objects this is "P6opaque".
This is basically the representation used by
classand its attribute specifications.
The NativeCall module provides a number or alternate memory representations, such as CStruct, CPointer and Cunion. Native arrays also have a different representation (VMArray).
say 42.REPR; # P6opaque
my int @a;
say @a.REPR; # VMArray
When a class is defined, it gets the P6opaque representation by default.
class Foo { }
say Foo.REPR; # P6opaque
Unless one is doing very deep core-ish work, how an object is represented in memory should not be of concern to you.
The DEFINITE method returns either True or False depending on whether the invocant has a concrete representation. This is almost always the same as calling the defined method. But in some cases it makes more sense in Raku to return the opposite with the defined method.
An example of this is the Failure class:
say Failure.new.DEFINITE; # True
say Failure.new.defined; # False
In general the defined method should be used. The DEFINITE method is intended to be used in very low-level (core) code. It's not all uppercase for nothing!
The reason
Failure.new.definedalways returnsFalseis to make it compatible withwith.
All of the introspection "metamethods" described in this blog post are actually parsed as macros directly generating low-level execution opcodes. This is really necessary in some cases (as otherwise information can be lost), and in other cases it's just for performance.
This doesn't mean that it's not possible to call this introspection functionality as a method: you can. For example:
my $a;
say "macro: " ~ $a.VAR.name; # macro: $a
say "method: " ~ $a."VAR"().name; # method: $a
Furthermore for consistency, the same functionality of these methods is also available as subroutines.
my $b;
say "sub: " ~ VAR($b).name; # sub: $b
So if you're more at home in imperative programming, you can do that as well!
This concludes the thirteenth episode of cases of UPPER language elements in the Raku Programming Language, the sixth episode discussing interface methods.
In this episode the following macro-like introspection methods were discussed (in alphabetical order): DEFINITE, HOW, REPR, VAR, WHAT, WHERE, WHO.
Stay tuned for the next episode!
This is a third follow-up on Raku Resolutions series.
The third meeting was held on 21 February 2026 at 19:00 UTC. Apart from 3 Raku Steering Council members, only 1 other person attended. In the end, 7 issues were discussed within the allotted time (1 hour).
After some discussion, it became clear that this issue basically has gone stale in light of the new raku.org site. Richard Hainsworth will create a new issue, after which this one can be closed.
unit be too late?
After some discussion it was agreed that from a language design point of view, it shouldn't really matter when the unit occurs in the code, as long as there's only one of them. And that the rest of the source would be considered to be part of that scope.
It was agreed that the documentation will describe this future state with the caveat that this is currently not yet the case. Eric Forste agreed to make a PR accordingly, after which the issue can be closed.
The point of being able to specify an EXPORT sub before a unit would be technically correct, but not very useful in practice as an EXPORT sub usually needs to refer to objects/classes that have already been defined in the code (and thus should logically be positioned after the scope).
List
After some discussion it was decided that the current behaviour of returning Nil for List elements beyond the end is correct. And that returning a Failure for negative index values is the most consistent behaviour, especially in light of Array doing that as well. So the issue could be closed.
Consensus was that it would be nice to have all of these math / statistics methods, but that these should live in module land for the foreseeable future. So the issue could be closed.
It was recognized that there are indeed quite a few open issues in Rakudo (although much less than there have been in the past). However, there will always be open issues. And with the current size of the core team, the number of issues will not significantly reduce in the future (with the current rate of new issues coming in).
So it was decided to close the issue as "unresolvable".
.classify
It was decided that the issue has gone a bit stale, and thus ask the OP (Original Poster) whether this should stay open, especially since it was suggested that .classify / .categorize maybe should need an overhaul (at least internally).
die with newline for raising end-user errors?
After explaining the esoteric rule in Perl 5 with regards to die with a message with and without a new line, it was agreed that it would be nice to have such a feature. Especially since at least 2 of the attendees had been using a note $message; exit 1 sequence to achieve just that. Disadvantage to this is that such a sequence can not be caught by a CATCH.
After some discussion, a :no-backtrace named argument to die() was suggested, and that the issue could be closed if a Pull Requst had been created for this. This has since then been implemented in #6076, so the issue was closed.
The next meeting will be held at 7 March 2026 at 19:00 UTC (20:00 CET, 14:00 EST, 11:00 PST, 04:00 JST (22 Jan), 06:00 AEST (22 Jan)), and again at a one hour maximum. If not all of these issues have been resolved, they will be moved to a future meeting.
Since Jitsi is still working out so far, the next one will be held at the same URL: https://meet.jit.si/SpecificRosesEstablishAllegedly. The reason Jitsi was selected, is that it has proven to be working with minimal hassle for at least the Raku Steering Council meetings. As the only thing you need to be able to attend, is a modern browser, a camera, and a microphone. No further installation required.
A new set of issues has been selected by yours truly (in issue number order, oldest first):
rw parameterRat and fail on lossy coercion)without allow chaining?.raku should be replaced with a pluggable systemAny Raku Community member is welcome to these meetings. Do you consider yourself a Raku Community member? You're welcome. It's as simple as that.
But please make sure you have looked at the issues that will be discussed before attending the meeting. And if you already have any comments to make on these issues, make them with the issue beforehand.
The original contributors to these issues will also be notified (unless they muted themselves from these issues). We hope that they also will be able to attend.
If you consider yourself a Raku Community member, please try to attend! If anything, it will allow you to put faces to the names that you may be familiar with.
Hope to see you there!
This is part twelve in the "Cases of UPPER" series of blog posts, describing the Raku syntax elements that are completely in UPPERCASE.
This part will discuss the various aspects of containers in the Raku Programming Language.
This may come as a shock to some, but Raku really only knows about binding (:=) values. Assignment is syntactic sugar.
What?
Most of the scalar variables that you see in Raku, are really objects that have an attribute to which the value is bound when you assign to the variable. These objects are called Scalar containers, or just short "containers".
In short: assignment in Raku is binding to an attribute in a
Scalarobject. Grokking how containers work in Raku (as opposed to many other programming languages) is one of the rites of passage to becoming an experienced and efficient developer in Raku. It sure was a rite of passage for yours truly!
In a simplified representation (almost pseudocode) one can think of these Scalar objects like this:
class Scalar {
has $!value;
method STORE($new-value) {
$!value := $new-value
}
method FETCH() {
$!value
}
}
And an assignment such as:
my $a;
$a = 42;
can be thought of as:
my $a := Scalar.new;
$a.STORE(42);
and showing the value of a variable (as in say $a), can be thought of as:
say $a.FETCH;
In reality it's of course slightly more complicated, because there are such things as type constraints that can exist on a variable (such as my Int $a). And variables may have a default value (as in my $a is default(42)).
This type of information is kept in a so-called "container descriptor", which is considered to be an implementation detail (as in: don't depend on its functionality directly, the interface to it might change at any time).
Many shortcuts are made in implementing this, in the interest of performance. Because assignments are one of the very basic operations in a programming language. And you want those to be fast!
As a mental model, this is very useful for understanding quite a few constructs in Raku.
In Raku any value returned at the end of a block (or with the return statement) are de-containerized by default. This means that something like this:
my $a = 42;
sub foo() { $a }
foo() = 666; # Cannot modify an immutable Int (42)
will not work. What if you could return the container from a block? Then you could just assign to it! There are indeed two easy ways to return something with the container intact: the is rw trait:
my $a = 42;
sub foo() is rw { $a }
foo() = 666;
say $a; # 666
Alternately, one can use the return-rw statement.
my $a = 42;
sub foo() { return-rw $a }
foo() = 666;
say $a; # 666
This feature is for instance used if you want to assign to an array element. In part 9 of this series it was shown that the AT-POS method is what is being called by postcircumfix [ ]. In the core, both the postcircumfix [ ] operator as well as the underlying AT-POS method have this trait set for performance (as the return-rw statement has slightly more overhead).
If an array is initialized with values, it will create containers for each element and put the right value in the right place. So you cannot only fetch the values from there, but you can also assign to these containers. And it is also possible to bind such a container to another variable.
Binding to containers allows for features in Raku that are considered "magic" by some, and maybe "too magic" by others. They are part of common idioms in the Raku Programming Language.
For instance: incrementing all values in an array:
my @a = 1,2,3,4,5;
$_++ for @a;
say @a; # [2 3 4 5 6]
The topic variable $_ binds to the elements of the array in a for loop. So in each iteration it actually represents the container at that location and can thus be incremented.
People coming from other languages may think that
$_is a "reference" (or "pointer") to the actual memory location of the element in the array. This notion is incorrect in Raku. The topic variable$_is bound to the container of each element. The container object itself is completely agnostic as to where it lives. It's just a simple object that knows how toFETCHandSTORE.
Slices to arrays also return containers. For instance to increment only the first and last element in an array without knowing its size:
my @a = 1,2,3,4,5;
$_++ for @a[0, *-1];
say @a; # [2 2 3 4 6]
The slice [0, *-1] produces 2 containers (one for the first element (0) and one for the last (*-1). Only these containers get incremented, thus giving the expected result.
The same logic applies to the values in hashes.
my %h = a => 42, b => 666;
$_++ for %h.values;
say %h; # {a => 43, b => 667}
But binding containers can also be done directly in your code. A contrived example:
my $a = 42;
my $b := $a;
$b = 666;
say $a; # 666
Note that by assigning to $b, you're assiging to the container that lives in $a. Because the binding of $b to $a effectively aliased the container in $a.
In Raku it is possible to assign to elements in an array that do not exist yet.
my @a;
@a[3] = 42;
say @a; # [(Any) (Any) (Any) 42]
So there's no container yet the element at index 3. Yet it is possible to assign to it! How does that work?
The secret is really in the "container descriptor". So let's refine our pseudocode representation of the Scalar class:
class Scalar {
has $!descriptor;
has $!value;
method STORE($value) {
$!value := $!descriptor.process($value);
}
method FETCH() {
$!value
}
}
Note that storing a value has now become a little more complicated ($!descriptor.process($value) rather than just $!value). So the descriptor object's process method takes the value, does what it needs to do with it, and then returns it so that it can be bound to the attribute.
This descriptor object is typically responsible for remembering the name of the variable, the default type, perform type checking and keep a default value of a container. And any additional logic that is needed.
There are many different types of container descriptor classes in the Rakudo implementation, all starting with the
ContainerDescriptor::name. And all are considered to be implementation detail.
For instance, when an AT-POS method is called on a non-existing element in an array, a container with a special type of descriptor is created that "knows" to which Array it belongs, and at which index it should be stored. The same is true for the descriptor of the container returned by AT-KEY (as seen in part 10) which knows in which Hash it should store when assigned to, and what key should be used.
It's this special behaviour that allows arrays and hashes to really just DWIM.
These special descriptors of containers for arrays and hashes also introduce an action-at-a-distance feature that you may or may not like.
my @a;
my $b := @a[3];
say @a; # []
$b = 42;
say @a; # [(Any) (Any) (Any) 42]
Note that the element in the array was not initialized after the binding, but only after a value was assigned to $b. This behaviour was specifically implemented this way to prevent accidental auto-vivification. For example:
my %h;
say %h<a><b>:exists; # False
say %h; # {}
%h<a><b> = 42;
say %h; # {a => {b => 42}}
So even though %h<a> is considered to be an Associative because of the <b>, the :exists test will not create a Hash in %h<a>. Only after a value has been assigned does %h<a> and %h<a><b> actually spring to life.
This rather lengthy introduction / diversion was to make you aware of some of the underlying mechanics of containers. Because Raku supplies a full customizable class that allows you to create your own container logic: Proxy. And understanding what containers are about is helpful when working with that class.
Creation of such a container is quite easy: all you need to supply are a method for fetching the value, and a method for storing a value (very similar to the pseudocode representation at the start of this blog post). This is done with the named arguments FETCH and STORE. Creation of a Proxy object is usually done inside a subroutine for convenience. A contrived example:
sub answer is rw {
my $value = 42;
Proxy.new(
FETCH => method () {
$value
},
STORE => method ($new) {
say "storing $new";
$value = $new;
}
)
}
my $a := answer;
say $a; # 42
$a = 666; # storing 666
say $a; # 666
The careful reader will have noticed that the is rw attribute needs to be specified on the subroutine, otherwise the Proxy would be de-containerized on return. And that the result of calling the answer subroutine was bound (:=) instead of assigned, because assignment would also cause de-containerization and thus completely defeat the purpose of this exercise.
Because the code in the supplied methods closes over the lexical variable $value, that variable stays alive until the Proxy object is destroyed. So it offers an easy way to actually store the value for this Proxy object.
Inside the supplied methods you are completely free to put whatever code that you want. As an example how that could work in a module, the Hash::MutableKeys distribution was created. Another case of BDD (Blog Driven Development)!
You may have observed that the
Proxyobject does not allow for a descriptor. It was not considered to be needed, as you have all the flexibility you could possibly want. If you want one in yourProxyobjects, you can create one yourself.
This concludes the twelfth episode of cases of UPPER language elements in the Raku Programming Language, the fifth episode discussing interface methods.
In this episode containers were described, as well as the special Proxy class with its named arguments STORE and FETCH. This Proxy class provides a fully customizable container implementation.
Stay tuned for the next episode!
This is part eleven in the "Cases of UPPER" series of blog posts, describing the Raku syntax elements that are completely in UPPERCASE.
This part will discuss the interface method that can be provided in a class to handle calls to methods that do not actually exist in this class (or its parents).
Method dispatch in the Raku Programming Language is quite complicated, especially in the case of multi-dispatch. But the first step is really to see if there is any (multi) method with the given name.
Internally this is handled by the find_method method on the meta-object of the class (which is typically called with the .^method syntax on the class).
So for example:
# an empty class
class A { }
dd A.^find_method("foobar"); # Mu
dd A.^find_method("gist"); # proto method gist (Mu $:: |) {*}
Note that an empty class A (which by default inherits from the Any class) does not provide a method "foobar" (indicated by the Mu type object). But it does provide a gist method (indicated by the proto method gist (Mu $:: |) {*} representation of a Callable) because that is inherited from the Any class.
It's this logic that is internally used by the dispatch logic to link a method name to an actual piece of code to be executed.
If the dispatch logic can not find a method by the given name, then it will throw an X::Method::NotFound error. One could of course use a CATCH phaser to handle such cases (as seen in part 6 of this series).
But there's a better and easier way to handle method names that could not be found. That is, if you're interested in somehow making them less fatal.
If a class provides a FALLBACK method (either directly in its class, or by one of its base classes, or by ingestion of a role), then that method will be called whenever a method could not be found. The name of the method will be passed as the first argument, and all other arguments will be passed verbatim.
A contrived example in which a non-existing method returns the name of the method, but only if there were no arguments passed:
class B {
method FALLBACK($name) { $name }
}
say B.foo; # foo
say B.bar(42); # Too many positionals passed; expected 2 arguments but got 3
So is there something special about the FALLBACK method? No, its only specialty is when it is being called. So you can make it a multi method:
class C {
multi method FALLBACK($name) {
$name
}
multi method FALLBACK($name, $value) {
"$value.raku() passed to '$name'"
}
}
say C.foo; # foo
say C.bar(42); # 42 passed to 'bar'
On the other hand if you don't care about any arguments at all and just want to return Nil, you can specify the nameless capture | as the signature:
class D {
method FALLBACK(|) { Nil }
}
say D.foo; # Nil
say D.bar(42); # Nil
The Raku Programming Language allows hyphens in identifier names (usually referred to as "kebab case"). Many programmers coming from other programming languages are not really used to this: they are more comfortable with using underscores in identifiers ("Snake case").
Modern Raku programs usually use kebab case in identifiers. So it's quite common for programmers to make the mistake of using an underscore where they should have been using a hyphen. If such an error is made when writing a program, it will be a runtime error. Which can be annoying. In such a case, a FALLBACK method can be a useful thing to have if it could correct such mistakes.
This could look something like this:
class E {
method foo-bar() { "foobar" }
method FALLBACK($name, |c) {
my $corrected = $name.trans("_" => "-");
if self.^find_method($corrected, :no_fallback) -> &code {
code(self, |c)
}
else {
X::Method::NotFound.new(
:invocant(self), :method($name), :typename(self.^name)
).throw;
}
}
}
The "E" class has a method "foo-bar". And a method "FALLBACK" that takes the name of the method that was not found (and putting any additional arguments in the Capture "c").
Note that the "c" is just an idiom for the name of a capture. It could have any name, but "c" is nice and short. If you want to use a name that's more clear to you, then please do so. As long as you use the same name later on.
It then converts all occurrences of underscore to hyphen in the name and then tries to find a Callable for that name. If that is successful it will execute the code with any arguments that were given by flattening the |c capture. Otherwise it throws a "method not found" error.
The
:no_fallbackargument is needed to prevent the method lookup from producing the "FALLBACK" method if the method was not found. Otherwise the code would loop forever.
So now this code will work instead of causing an execution error.
say E.foo-bar; # foobar
say E.foo_bar; # foobar
To make this more generally usable this code could be put into a role and have that live as an installable module. But that would be extra work, wouldn't it?
Fortunately the writing of this blog post initiated the development of such a role (and associated distribution) called Method::Misspelt. How's that for BDD (Blog Driven Development)?
Note that the module introduces a few more features and optimizations, while handling keeping track of multiple classes ingesting the same role. See the internal documentation for more information.
This concludes the eleventh episode of cases of UPPER language elements in the Raku Programming Language, the fourth episode discussing interface methods.
In this episode the FALLBACK method was described, as well as some simple customizations. And a bonus module created for this blog post only.
Stay tuned for the next episode!
We needed a legal document with numbered Articles, and the document needed to be in MarkDown for github issues, and in html, so naturally the source should be in RakuDoc.
Although numbered headings and items were specified in RakuDoc v1 (aka POD6), they were never implemented. In RakuDoc v2 there was progress, and together with some tweaking of the numitem templates, it was easy to write the RakuDoc source.
However, when we were revising RakuDoc, Damian Conway had some extra ideas about generalising enumeration, including ideas about adding alias definitions so that the numbered block could be referenced later in the text.
Since it had been so easy to tweak the numitem templates, I thought it would be easy upgrade the specification of RakuDoc v2 to get these generalisations. Was I ever so wrong!!!! Ask a genius with decades of language design experience for a nice design and you get an effusion of ideas and extensions that make RakuDoc better than any editor I have ever used, but with a simplicity that makes it easy for a document author to understand.
Another result of the redesign is to make the underlying specification of RakuDoc much clearer, something I will cover later.
During the development process, my daughter mentioned that her friend had just finished a PhD dissertation and was complaining about how much time she needed to spend reformatting the text because the numbering kept getting out of sync. The problem with most editors is that most of the effort goes on perfecting the user-facing interface, while the underlying format is created ad hoc, and enumeration is an addition. Getting the underlying structure right will make subsequent rendering easier.
Just as the design was ending and the renderer passing most tests, I casually mentioned citations would be a good extension. The "standards" for citations number in the thousands! .oO(The Jabberwocky ) But Damian snicker-snacked his vorpal blade, reducing the monstrous tangle to something easier to use. RakuDoc v2 now has a =citation block and Q<> markup to insert quoted citations. I will cover these additions in the next blog (as in: when I get the renderer to work with the new ideas).
Back to enumerating RakuDoc, here are some examples to illustrate the new functionality.
Suppose you have a formula (or code or map or table or item in a list) and you want to number it, and then reference the number in the text? Then another formula gets added into the text before, well you would want the references to update as well.
Suppose you have tables that you want to be enumerated separately from headings? But also by preference you want them to be enumerated in sequence with headings? That is the prefix to the table enumeration is the heading number.
Suppose you want Chinese, Roman, or Bengali numbering?
AND suppose you also want the numbering to be a mix of numerals and other characters, such as brackets? This is a common requirement in legal documents.
Suppose you want Arbitrary words before after or in between the numbers? For example, Article 1., Article 2. etc.
Suppose you want to have the paragraphs numbered? Sometimes you might want the paragraphs numbered in sequence from the start of the document, and sometimes you want the numbering to restart after a new heading.
Suppose you want to have the Tables and the Formulae numbered in sequence? Or perhaps, for a section that explains some aspect of one formula, you want to number the formulae separately, but then return to the original sequence after that section?
The updated RakuDoc v2 allows for all of these possibilities, while also providing for sensible default option values.
Moreover, RakuDoc allows for custom blocks, eg. LeafletMap - a map block that exposes the marvelous Leaflet library. Now it is an automatic part of the specification that prefixing a custom block with num (numLeafletMap) will number the caption of the block.
An HTML rendering of the new RakuDoc v2 specification containing the enumerated functionality can be found at RakuDoc enumeration branch
In the specification of RakuDoc, there are discussions about directives, blocks, metaoptions, and a section on =config.
Since the syntax for a directive is almost the same as the syntax of a block, it is not immediately apparent what the difference is. As we developed the generic enumeration, it became much clearer what the difference is.
Furthermore, the older specification covered multi-level headings and items. This functionality is now extended to all blocks, so =numtable2 or =numcode3 will be numbered in parts from the previous instance of =table (the equivalent of table1) or =code. This means that we need to carefully distinguish between the block base (eg., table or code), and the block level (eg., 1, 2, 3). Together the base and the level create a blocktype. These distinctions, although implied, were not so clear in the older specification.
Let's return to the =config directive. The first parameter of =config is the name of a blocktype. Note that the num prefix is not a part of the blocktype and only indicates that the enumeration associated with the instance needs to be rendered. Consequently, the presence or absence of 'num' in the config parameter has no significance. The next =config parameters are all, and may only be, metadata options.
The function of the =config directive is to distribute the named metadata options, and their values, to the named blocktype. In addition, the same metadata option can be specified on a blocktype instance (using either the =for or =begin directives), in which case it takes precedence over the option in the config. A consequence of this paradigm is that each metadata option has to have a semantic significance in the context of a blocktype instance.
One of the design aims was to give a document author the choice of restarting the enumeration for some blocktype when another blocktype is encountered. For example, the original specfication for =numitem included the idea that whenever a sequence of =numitems was encountered, they would form an ordered set, and that if another block, such as a paragraph, was encountered, the enumeration would be restarted.
This means that the occurrence of a =head restarts the item counter. This cannot be controlled within the handler of a block. Consequently, any option affecting block counting needs to be distinct from the rendering of the blocks themselves.
After some design iterations, a new directive called =counter was introduced. A directive, as opposed to a block, affects all subsequent blocks in the RakuDoc source, within the same scope. A block, and the metadata options operating on the blocktype, only affects its immediate contents.
Further, it became clear that a block and its counter were different objects, although for simplicity they have the same name. For most needs, an author does not need to know that a counter and a block are different, except that when a new counter is needed, the difference becomes necessary. In a similar way, for many purposes it doesn't matter whether a number is an integer or a string, except when it does. Raku allows for things to become complicated when the author needs it to be.
To make it easier to experiment with RakuDoc and the enumerated functionality, I have developed a Docker image called browser-editor. It is based on an Alpine image and contains raku, Cro, and the latest version of Rakuast::RakuDoc::Render (currently not yet in the fez system).
Using podman, the image can be put into a container and run locally on Linux based systems, thus:
podman pull docker.io/finanalyst/browser-editor:latest
podman run -d -v .:/browser/publication --rm -name rb docker.io/finanalyst/browser-editor:latest
For Mac silicon, a small change is needed to indicate the platform inside the docker image is based on Linux, thus:
podman run -d -v .:/browser/publication --rm --platform linux/amd64 --name rb docker.io/finanalyst/browser-editor:latest
In both cases the container is given the arbitrary name rb, and so when it is time to stop the container, the following is sufficient:
podman stop rb
The directory that the container is started from will then linked to the container's /browser/publication directory, and changes and new RakuDoc source files will be saved in that directory.
A RakuDoc source can then be edited and the HTML rendering is created on the fly, by pointing a browser at (setting the browser URL to) localhost:3000.
You should see something like the following in the browser:
Two frameworks exist: one needing no internet connection - a minimal single file rendering, and another that uses plugins to expose some useful third-party libraries and the more sophisticated Bulma CSS framework. The choice is toggled using the 'online' button.
Try selecting 'online' and copying in the following test source by Damian Conway to see some different effects.
=begin rakudoc
=TITLE Taster source
=numitem One
=numitem Two
=counter item :restart(42)
=numitem Three
=counter item :restart
=numitem Four
=counter item :prefix<head3>
=numitem Five
=counter item :!restart
=para ???
=numitem Six
=numpara
S<Now is the winter of our "no content"
Made glorious summer by Camelia’s bloom;
And all backlog that lour’d upon our docs
In deep commits is buried, link’d, and tagged.
Our nightly builds now wear triumphant green,
Our failing tests to passing smiles are turn’d;
Grim sighs of “ere next Yule” have chang’d to grins,
And hacker dread to blog posts boldly strung.>
=numpara
S<But I—long nurs’d on RFCs and hope,
Deform’d by specs that shifted as I read,
Unfit for idle scripts or stable sleep—
Am set, since long delays are now no more,
To ship Raku...and break the world anew.>
=numcode
$x = any <1 2 3>;
=for numcode :caption<Same thing, just labelled>
$x = any <1 2 3>;
=for numformula :caption<This means nothing!>
x^2 = y_j + \sum z_i
=for numformula :caption<This means nothing!> :alt< xH<2> = yJ<1> + E<GREEK CAPITAL LETTER SIGMA> zJ<i> >
x^2 = y_j + \sum z_i
=for numformula :alt< (1+x)H<n> = E<GREEK CAPITAL LETTER SIGMA> H<n>CJ<i> xH<i> > :caption<This is actually right>
(1+x)^n = \sum_{i=0}^n {n \choose i} x^i
=for numinput
Hey type something:
=numoutput
You typed: hsdkhldkhskdhasd
=counter numtable :restart(33456)
=begin numtable :caption<Encryption table> :form<Example %R: %C:pc>
=row
=cell A
=cell B
=cell C
=row
=cell D
=cell E
=cell F
=end numtable
=end rakudoc
This should look like the following (or at least the top part).
The example shows some features like numbering paragraphs, code examples and tables. But you will also see the difference between the two rendering contexts. The online version has access to an online latex rendering resource, so the formula are nicely rendered, whilst the off-line version only renders to the raw formula or a character-based alternative - if provided.
Suppose we want to enumerate a code sample and then refer to it later, we can do the following:
=begin rakudoc :!toc
=TITLE Enumerations
=for numcode :numalias<SAY_EX> :lang<raku>
my %h = <one two three> Z=> ^3;
say %h;
=numoutput {one => 0, three => 2, two => 1}
=for numcode :numalias<PUT_EX> :lang<raku>
my %h = <one two three> Z=> ^3;
put %h;
=for numoutput
one 0
three 2
two 1
The difference between A<SAY_EX> and A<PUT_EX> is that C<say> and C<put> use different methods to convert the C<%h> structure into printable strings.
=end rakudoc
In the browser-editor image, we will get an HTML rendering something like:
The
=begin rakudoc :!tocand=end rakudocare used to top and tail the RakuDoc example above because the HTML renderer expects a complete Raku program, and a RakuDoc source (currently) needs the RakuDoc to be within a=rakudocblock. The:!tocswitches off the automatic Table of Contents that the HTML utility of theRakuast::RakuDoc::Renderdistribution generates. (Try removing:!toc)
Suppose we want out tables to have enumerations that align with the enumerations of the headings. So this means we are requiring a different sort of behaviour from the counter associated with the table block base. Consequently, we need to use the =counter directive. Also note that we are prefixing with the head2 block, which has an implicit prefix of head1.
=begin rakudoc :!toc
=TITLE Prefixes
=counter table :prefix<head>
=numhead First title
=numhead2 Sub first title
=for numtable :caption<First table>
| one | two |
=for numtable :caption<Second table>
| three | four |
=numhead Second Title
=counter table :restart(6)
=for numtable :caption<Third table>
| five | six | seven | eight |
=for numtable2 :caption<Subordinate table>
| nine | ten |
| nine | ten |
=end rakudoc
Comments:
table counter to 6 before Third table. 
To illustrate the multi-lingual ability of RakuDoc (and Raku in general), lets use Chinese, Roman and Bengali for multilevel headings.
=begin rakudoc :!toc
=TITLE Some non-Arabic numerals
=config head :form< %Z %D >
=config head2 :form< ｢%Z｣%R %D >
=config head3 :form< ｢%Z｣%R｢%B｣ %D >
=numhead First title
=numhead Second title
=numhead3 Sub-sub-head one (implies the sub-head)
=numhead3 Sub-sub-head two
=numhead2 Sub-head first explicit
=numhead2 Sub-head second explicit
=for numformula2 :form<%T %Z.%R. >
\begin{align*}
\sum_{i=1}^{k+1} i^{3}
&= \biggl(\sum_{i=1}^{n} i^{3}\biggr) + i^3\\
&= \frac{k^{2}(k+1)^{2}}{4} + (k+1)^3 \\
\end{align*}
=end rakudoc
This will produce an HTML rendering a bit like
The :numform option is very flexible and in fact the brackets in the previous example are arbitrary text. The following RakuDoc source
=begin rakudoc :!toc
=TITLE Arbitrary text in enumeration
=config head :form< Article %N. %D >
=config head2 :form< Article %N.%r. %D >
=config head3 :form< Article %N.%r｢%Z｣ %D >
=numhead First title
=numhead Second title
=numhead3 Sub-sub-head one (implies the sub-head)
=numhead3 Sub-sub-head two
=numhead2 Sub-head first explicit
=numhead2 Sub-head second explicit
=end rakudoc
will produce something like:
In RakuDoc, paragraphs as in all text editors, are strings of text ending in a blank line or a new block (for RakuDoc). There is no need to mark a paragraph block, but it will be equivalent to a block marked =para.
By default, each paragraph is numbered from the start of the document. In order to see the enumeration, the paragraph does need to be started with a =numpara.
For this example, we want to number paragraphs in each section marked with a heading. So we need to tell the counter which conditions it is restarted by; in this case after every =head (by default a bare block name has a level of 1). There are both :restart-after and :restart-except-after conditions, but that's all we'll say about them here.
The RakuDoc source is
=begin rakudoc :!toc
=TITLE Restarting after headings
=counter para :restart-after<head>
=head First heading
=numpara First line
=numpara Second line
=numpara Third line
=head Second heading
=numpara First line 2
=numpara Second line 2
=head2 Subordinate heading does not restart the paragraphs
=numpara Third line 2
=numpara Fourth para
=end rakudoc
The source will yield something like:
We want to number code and formulae using the same counter, but also we want to explain some formula with a special numbering that is then forgotten.
Since all =config and =counter statements are scoped, all we need to do to get the special numbering is to create a =section, which introduces a new block scope. The previous config/counter options continue, unless explicitly overridden.
However, it would not be useful if upon exiting a section, the numbering of a counter reverts to the value before the section. So there is a subtle distinction between the counter and the value of the count it contains. The counter options, such as its prefix, and when it is restarted, are scoped, but the value of the counter is not scoped. In order to reset the counter value, the counter has to be reset.
RakuDoc v2 clarified the idea of custom blocks. Previously it was implied that developers could have their own blocks, but in V.2 the difference between built in and custom blocks was clarified: custom blocks must contain a mix of upper and lower case letters (more precisely characters with the Unicode properties Lu and Ll).
The enumeration options use this idea by creating custom counters. Since this is something that managed by a block, the :counter option is provided to the appropriate blocks using a =config directive.
=begin rakudoc :!toc
=TITLE Using custom counnters
=config formula :counter< FormulaeTables >
=config table :counter< FormulaeTables >
=numhead First title
=numtable
| one | two | three |
=numtable
| more | tabula data |
| 5 | 6 |
| 7 | 8 |
=numformula E = mc^2
=numformula e^{\pi i} = -1
=begin section
=config table :counter< Derivation > :form<Aside: %T %N>
=numhead2 Some explanation
=numtable
| this | data | is | more | detailed |
| 1 | 2 | 3 | 4 | 5 |
=end section
=numhead Continuing
=numtable
| here | we | resume |
=end rakudoc
This will produce something like
We have only just completed the revised specification, and the Renderer has only just been developed. It is inevitable that there will be flaws, and the styling choices may not be liked by everyone.
Even so, we think this new upgrade of RakuDoc v2 will prove to have a much wider applicability than just a documentation aid to Raku programs.
The docker image allows anyone to experiment with the new RakuDoc functionality immediately without needing to install the whole Raku / Cro / Rakuast::RakuDoc::Render stack and some dependencies, such as Dart Sass, locally.
Dsci is a brand new kid on the cicd area. It allows use of general programming languages to create ci scenarios.
——-
Consider some imaginary case of building and deploying an application as docker container. So, logically we have two main stages:
build docker image from source code
deploy docker image using docker pull and docker run commands
Let’s see how dsci could tackle such a scenario …
——-
The main concept we start with is so called ‘pipeline file’.
It’s just yaml file containing a list of jobs to be performed. Every job is executed on isolated docker container, execution logic is always sequential - one job, by another.
This pattern yet simple however covers the majority of use cases:
jobs.yaml
jobs:
-
id: build
path: build/
-
id: deploy
path: deploy/
Thus, every element of job list is a job, where job has some unique identifier and path to location of job file.
Usually there are some conditions when and what ci scripts are triggered. Those conditions may be based on different criteria, but usually they depend on branch names.
Dsci utilizes a special syntax to define those rules. Let’s say we only want to deploy if source branch is main, and perform build for all branches:
jobs:
-
id: build
path: build/
-
id: deploy
path: deploy/
only: .<ref> eq "refs/heads/main"
Read more on job conditions on dsci documentation - https://github.com/melezhik/DSCI/blob/main/job-cond.md
To pass parameters to job, just use params: key:
jobs:
-
id: build
path: build/
params:
foo: bar
version: 0.1.0
-
id: deploy
path: deploy/
Those parameters could be thought as overrides for default ones, however dsci allows more then that - pass results ( states ) from one job to another, see later.
The next important building block of the hierarchy is so called “job file” defining main job logic.
Job file needs to be named depending on language of choice, in our example we use Bash as it usually is enough for many use cases. But using other languages is also possible. If one needs more flexibility they may chose Python or Golang
build/task.bash
version=$(config version)
tag_version=$(date +%s).$version
docker build . -t app:$tag_version
docker push repo/app:$tag_version
In this simple example our job file is just a single task job. However if there is a need to split complex job into several tasks they may do so by using job.$ext file approach. This file needs to be named according language of choice.
Let’s say we have three tasks - configure, build and test - incorporated into build job, we may run them one by one like this:
build/job.py
#!/bin/python
run_task(“configure”);
run_task(“build”);
run_task(“test”);
And if we put those tasks under build/tasks/ directory like this:
build/tasks/configure/task.bash
build/tasks/build/tasks.bash
build/tasks/tests/task.bash
In this case we have modular setup of our ci job.
The neat thing about this DSL, dsci provides the same SDK for all supported programming languages.
Read more about jobs and tasks on dsci documentation web site:
Let’s make our last job example more realistic and return docker image tag dynamically created by build job back to deploy job. We may want to do so deploy jobs know what tag to pull before deploy. All we need is to add extra line to save the job state into internal dsci cache:
build/task.bash
version=$(config version)
tag_version=$(date +%s).$version
docker build . -t app:$tag_version
docker push repo/app:$tag_version
update_state tag $tag_version
update_state() is very handy function allowing to pass states between different tasks and jobs. It’s implemented for all supported languages.
To pick up tag name in deploy job we can use already mentioned config() function:
deploy/task.bash
set -e
tag=$(config tag)
docker pull repo/app@$tag
docker stop -t 1 container || :
docker run -rm -name container -td repo/app@$tag
Read more about job and tasks states on dsci documentation ( the links above ^^ )
Let’s modify build job example with pushing image to docker registry. To do a push we need to authenticate against a docker registry first. Dsci enables simple way to pass secrets to pipelines.
This time let’s rewrite our job file on Python for convenience:
build/job.py
#!/bin/python
import os
password = os.environ['password'])
run_task(“build”, { “password”: password })
And then modify build task to handle password parameter:
version=$(config version)
tag_version=$(date +%s).$version
docker build . -t app:$tag_version
update_state tag $tag_version
echo $password | docker login --username your-username --password-stdin
docker push repo/app:$tag_version
As we can see secrets passed into pipelines as environment variables.
And unlike other task and jobs parameters they never saved in job reports or cache files.
—-
By default dsci pipelines run inside some docker container, this fits situation when one needs run purely ci code - for example build and run some unit tests. CD part comes to play when build artifacts are ready for deploy.
Dsci allows switch to deployment environment by using localhost switcher:
jobs:
-
id: build
path: build/
params:
foo: bar
version: 0.1.0
-
id: deploy
path: deploy/
localhost: true
In that case deployment occurs on VM running dsci orchestrator and which allows to restart docker container with new image version right on VM
——-
That is it.
This simple but real life example shows how easy and one can write cicd pipelines using dsci framework and how flexible it is.
Hopefully you like it.
—-
For comprehensive documentation and more information please visit dsci web site - http://deadsimpleci.sparrowhub.io/doc/README
This is part ten in the "Cases of UPPER" series of blog posts, describing the Raku syntax elements that are completely in UPPERCASE.
This part will discuss the interface methods that one can implement to provide a custom Associative interface in the Raku Programming Language. Associative access is indicated by the postcircumfix { } operator (aka the "hash indexing operator").
In a way this blog post is a cat license ("a dog licence with the word 'dog' crossed out and 'cat' written in crayon") from the previous post. But there are some subtle differences between the
Positionaland theAssociativeroles, so just changing all occurrences of "POS" in "KEY" will not cut it.
The Associative role is really just a marker, just as the Positional role. It does not enforce any methods to be provided by the consuming class. So why use it? Because it is that constraint that is being checked for any variable with a % sigil. An example:
class Foo { }
my %h := Foo;
will show an error:
Type check failed in binding; expected Associative but got Foo (Foo)
However, if we make that class ingest the Associative marker role, it works:
class Foo does Associative { }
my %h := Foo;
say %h<bar>; # (Any)
and it is even possible to call the postcircumfix { } operator on it! Although it doesn't return anything really useful.
Note that the binding operator
:=was used: otherwise it would be interpreted as initializing a hash%hwith a single value, which would complain with an "Odd number of elements found where hash initializer expected: Only saw: type object 'Foo'" execution error.
The postcircumfix { } operator performs all of the work of slicing and dicing objects that perform the Associative role, and handling all of the adverbs: :exists, :delete, :p, :kv, :k, and :v. But it is completely agnostic about how this is actually done, because all it does is calling the interface methods that are (implicitely) provided by the object, just as with the Positional role. Except the names of the interface methods are different. For instance:
say %h<bar>;
is actually doing:
say %h.AT-KEY("bar");
under the hood. So these interface methods are the ones that actually know how to work on an Hash, Map or a PseudoStash, or any other class that does the Associative role.
Let's introduce the cast of this show (the interface methods associated with the Associative role):
The AT-KEY method is the most important method of the interface methods of the Associative role: it is expected to take the argument as a key, and return the value associated with that key.
Note that the key does not need to be a string, it could be any object. It's just that the implenentation of
HashandMapwill coerce the key to a string.
The AT-KEY method should return a container if that is appropriate, which is usually the case. Which means you probably should specify is raw on the AT-KEY method if you're implementing that yourself.
say %h<bar>; # same as %h.AT-KEY("bar")
The EXISTS-KEY method is expected to take the argument as a key, and return a Bool value indicating whether that key is considered to be existing or not. This is what is being called when the :exists adverb is specified.
say %h<bar>:exists; # same as %h.EXISTS-KEY("bar")
The DELETE-KEY method is supposed to act very much like the AT-KEY method. But it is also expected to remove the key so that the EXISTS-KEY method will return False for that key in the future. This is what is being called when the :delete adverb is specified.
say %h<bar>:delete; # same as %h.DELETE-KEY("bar")
The ASSIGN-KEY method is a convenience method that may be called when assigning (=) a value to a key. It takes 2 arguments: the key and the value, and is expected to return the value. It functionally defaults to object.AT-KEY(key) = value. A typical reason for implementing this method is performance.
say %h<bar> = 42; # same as @a.ASSIGN-KEY("bar", 42)
The BIND-KEY method is a method that will be called when binding (:=) a value to a key. It takes 2 arguments: the key and the value, and is expected to return the value.
say %h<bar> := 42; # same as %h.BIND-KEY("bar", 42)
The STORE method accepts the values (as an Iterable) with which to (re-)initialize the hash, and returns the invocant.
The :INITIALIZE named argument will be passed with a True value if this is the first time the values are to be set. This is important if your data structure is supposed to be immutable: if that argument is False or not specified, it means a re-initialization is being attempted.
say %h = a => 42, b => 666; # same as %h.STORE( (a => 42, b => 666) )
Although not an uppercase named method, it is an important interface method: the keys method is expected to return the keys in the object.
my %h = a => 42, b => 666;
say %h.keys; # (a b)
Again, that's a lot to take in (especially if you didn't read the previous post)! But if it is just a simple customization you wish to do to the basic functionality of e.g. a hash, you can simply inherit from the Hash class. Here's a simple, contrived example that will return any values doubled as string:
class Hash::Twice is Hash {
method AT-KEY($key) { callsame() x 2 }
}
my %h is Hash::Twice = a => 42, b => 666;
say "$_: %h{$_}" for %h.keys;
will show:
a: 4242
b: 666666
Note that in this case the callsame function is called to get the actual value from the array.
If you want to be able to do more than just simple modifications, but for instance have an existing data structure on which you want to provide an Associative interface, it becomes a bit more complicated and potentially cumbersome.
Fortunately there are a number of modules in the ecosystem that will help you to create a consistent Associative interface for your class.
The Hash::Agnostic distribution provides a Hash::Agnostic role with all of the necessary logic for making your object act as a Hash. The only methods that must be provided, are the AT-KEY and keys methods. Your class may provide more methods for functionality or performance reasons.
The Map::Agnostic distribution provides a Map::Agnostic role with all of the necessary logic for making your object act as a Map. The only methods that you must be provided, are the AT-KEY and keys methods. As with Hash::Agnostic, your class may provide more methods for functionality or performance reasons.
A very contrived example in which a Hash::Int class is created that provides an Associative interface in which the keys can only be integer values. Under the hood it uses an array to store values:
use Hash::Agnostic;
class Hash::Int does Hash::Agnostic {
has @!values;
method AT-KEY(Int:D $index) {
@!values.AT-POS($index)
}
method keys() {
(^@!values).grep: { @!values.EXISTS-POS($_) }
}
method STORE(\values) {
my @values;
for Map.CREATE.STORE(values, :INITIALIZE) {
@values.ASSIGN-POS(.key.Int, .value)
}
@!values := @values;
}
}
and can be used like this:
my %h is Hash::Int = <42 a 666 b 137 c>;
say %h<42>;
dd %h;
which would show:
a
Hash::Int.new(42 => "a",137 => "c",666 => "b")
Again, note that the STORE method had to be provided by the class to allow for the is Hash::Int syntax to work.
If you're wondering what's happening with
Map.CREATE.STORE(values, :INITIALIZE): the initialization logic of hashes and maps allows for both separate key,value initialization, as well as key=>value initialization. And any mix of them. So this is just a quick way to use that rather specialized logic ofMap.newto create a consistentSeqofPairs with which to initialize the underlying array.Raku actually has a syntax for creating a
Hashthat only takesIntvalues as keys:my %h{Int}. This creates a so-called "object hash" with different performance characteristics to the approach taken withHash::Int.
This concludes the tenth episode of cases of UPPER language elements in the Raku Programming Language, the third episode discussing interface methods.
In this episode the AT-KEY family of methods were described, as well as some simple customizations and some handy Raku modules that can help you create a fully functional interface: Map::Agnostic and Hash::Agnostic.
Stay tuned for the next episode!
Every year on February 7th, math enthusiasts worldwide (should) consider celebrating Euler’s Day or E-day. Among Euler’s many gifts to the (currently known) mathematical universe is the ever-popular number e, the natural logarithm base that is basically the rock star of calculus, complex analysis, continuous growth models, compound interest, and (much) more. That irrational number shows up in places we might or might not expect. This blog post (notebook) explores some formulas and plots related to Euler’s number, e.
Remark: The code of the fractal plots is Raku translation of the Wolfram Language code in the notebook “Celebrating Euler’s day: algorithms for derangements, branch cuts, and exponential fractals” by Ed Pegg.
use JavaScript::D3;use JavaScript::D3::Utilities;
#% javascriptrequire.config({ paths: { d3: 'https://d3js.org/d3.v7.min'}}); require(['d3'], function(d3) { console.log(d3);});
#% jsjs-d3-list-line-plot(10.rand xx 40, background => 'none', stroke-width => 2)
my $title-color = 'Silver';my $background = '#1F1F1F';
Raku has the built in mathematical constant (base of the natural logarithm). Both ASCII “e” and Unicode “𝑒” (“MATHEMATICAL ITALIC SMALL E” or U+1D452) can be used:
[e, 𝑒]
# [2.718281828459045 2.718281828459045]
We can verify this famous equation:
e ** (i * π) + 1
# 0+1.2246467991473532e-16i
Let us compute using the canonical formula:
Here is the corresponding Raku code:
my @e-terms = ([\*] 1.FatRat .. *);my $e-by-sum = 1 + (1 «/» @e-terms[0 .. 100]).sum
# 2.7182818284590452353602874713526624977572470936999595749669676277240766303535475945713821785251664274274663919320030599218174135966290435729003342952605956307381312
Here we compute the e using Wolfram Language (via wolframscript):
my $proc = run 'wolframscript', '--code', 'N[E, 100]', :out;my $e-wl = $proc.out.slurp(:close).substr(0,*-6).FatRat
# 2.7182818284590452353602874713526624977572470936999595749669676277240766303535475945713821785251664274274661651602106
Side-by-side comparison:
#% html[ {lang => 'Raku', value => $e-by-sum.Str.substr(0,100)}, {lang => 'Wolfram Language', value => $e-wl.Str.substr(0,100)}]==> to-html(field-names => <lang value>, align => 'left')
| lang | value |
|---|---|
| Raku | 2.71828182845904523536028747135266249775724709369995957496696762772407663035354759457138217852516642 |
| Wolfram Language | 2.71828182845904523536028747135266249775724709369995957496696762772407663035354759457138217852516642 |
And here is the absolute difference:
abs($e-by-sum - $e-wl).Num
# 2.2677179245992183e-106
Let us next compute e using the continued fraction formula:
To make the corresponding continued fraction we first generate its sequence using Philippe Deléham formula for OEIS sequence A003417:
my @rec = 2, 1, 2, 1, 1, 4, 1, 1, -1 * * + 0 * * + 0 * * + 2 * * + 0 * * + 0 * * ... Inf;@rec[^20]
# (2 1 2 1 1 4 1 1 6 1 1 8 1 1 10 1 1 12 1 1)
Here is a function that computes the continuous fraction formula:
sub e-by-cf(UInt:D $i) { @rec[^$i].reverse».FatRat.reduce({$^b + 1 / $^a}) }
Remark: A more generic continued fraction computation is given in the Raku entry for “Continued fraction”.
Let us compare all three results:
#% html[ {lang => 'Raku', formula => 'sum', value => $e-by-sum.Str.substr(0,100)}, {lang => 'Raku', formula => 'cont. fraction', value => &e-by-cf(150).Str.substr(0,100)}, {lang => 'WL', formula => '-', value => $e-wl.Str.substr(0,100)}]==> to-html(field-names => <lang formula value>, align => 'left')
| lang | formula | value |
|---|---|---|
| Raku | sum | 2.71828182845904523536028747135266249775724709369995957496696762772407663035354759457138217852516642 |
| Raku | cont. fraction | 2.71828182845904523536028747135266249775724709369995957496696762772407663035354759457138217852516642 |
| WL | – | 2.71828182845904523536028747135266249775724709369995957496696762772407663035354759457138217852516642 |
The maximum of the function x^(1/x) is attained at e:
#% jsjs-d3-list-line-plot((1, 1.01 ... 5).map({ [$_, $_ ** (1/$_)] }), :$background, stroke-width => 4, :grid-lines)
The Exponential spiral is based on the exponential function (and below it is compared to the Archimedean spiral):
#% jsmy @log-spiral = (0, 0.1 ... 12 * π).map({ e ** ($_/12) «*» [cos($_), sin($_)] });my @arch-spiral = (0, 0.1 ... 12 * π).map({ 2 * $_ «*» [cos($_), sin($_)] });my %opts = stroke-width => 4, :!axes, :!grid-lines, :400width, :350height, :$title-color;js-d3-list-line-plot(@log-spiral, :$background, color => 'red', title => 'Exponential spiral', |%opts) ~js-d3-list-line-plot(@arch-spiral, :$background, color => 'blue', title => 'Archimedean spiral', |%opts)
Catenary is the curve a hanging flexible wire or chain assumes when supported at its ends and acted upon by a uniform gravitational force. It is given with the formula:
Here is a corresponding plot:
#% jsjs-d3-list-line-plot((-1, -0.99 ... 1).map({ [$_, e ** $_ + e ** (-$_)] }), :$background, stroke-width => 4, :grid-lines, title => 'Catenary curve', :$title-color)
The exponential curlicue fractal:
#%jsjs-d3-list-line-plot(angle-path(e <<*>> (1...15_000)), :$background, :!axes, :400width, :600height)
Here is a plot of exponential Mandelbrot set:
my $h = 0.01;my @table = do for -2.5, -2.5 + $h ... 2.5 -> $x { do for -1, -1 + $h ... 4 -> $y { my $z = 0; my $count = 0; while $count < 30 && $z.abs < 10e12 { $z = exp($z) + $y + $x * i; $count++; } $count - 1; }}deduce-type(@table)
#% jsjs-d3-matrix-plot(@table, :!grid-lines, color-palette => 'Rainbow', :!tooltip, :!mesh)
A fractal variant using reciprocal:
my $h = 0.0025;my @table = do for -1/2, -1/2 + $h ... 1/6 -> $x { do for -1/2, -1/2 + $h ... 1/2 -> $y { my $z = $x + $y * i; my $count = 0; while $count < 10 && $z.abs < 100000 { $z = exp(1 / $z); $count++; } $count; }}deduce-type(@table)
#% jsjs-d3-matrix-plot(@table, :!grid-lines, color-palette => 'Rainbow', :!tooltip, :!mesh)
DTAP is a new testing protocol allowing to test infrastructure with just a Bash scripts. Here is a quick example, let's test that /etc/dhcp/ directory and /etc/dhcpcd.conf file exists:
#!/bin/bash
session=$(date +%s)
ls /etc/dhcp/ 2>&1 | dtap --box - \
--session $session \
--params path=/etc/dhcp/ \
--check path-ok \
--desc "dhcp/ dir"
ls /etc/dhcpcd.conf 2>&1 | dtap --box - \
--session $session \
--params path=/etc/dhcpcd.conf \
--check path-ok \
--desc "dhcpcd.conf file"
dtap --report --session $session
The result will be:
DTAP report
session: 1767875428
===
dhcp/ dir ...... OK
dhcpcd.conf file ...... OK
Plain and simple
As one can see test scripts are just plain Bash commands, no fancy YAML or even high level programming languages.
Also DTAP follows WYSIWYG principle when we get exactly what we see, in a sense this is something we would do trying to check existence of the mentioned directory and file:
ls /etc/dhcp/
ls /etc/dhcpcd.conf
And if any errors occurr we will get exactly what we are asking for - output of ls command which most of the Linux users probably are familiar with:
!/bin/bash
ls /etc/does-not-exist 2>&1 | dtap --box - \
--session $session \
--params path=/etc/does-not-exist \
--check path-ok \
--desc "/etc/does-not-exist dir"
Output:
DTAP report
session: 1767875841
===
/etc/does-not-exist dir ...... FAIL
[report]
15:37:22 :: [sparrowtask] - run sparrow task .@path=/etc/does-not-exist
15:37:22 :: [sparrowtask] - run [.], thing: .@path=/etc/does-not-exist
[task run: task.bash - .]
[task stdout]
15:37:23 :: ls: cannot access '/etc/does-not-exist': No such file or directory
[task check]
stdout match <^^ "/etc/does-not-exist" $$> False
=================
TASK CHECK FAIL
2
How this works?
There are two essentials primitives in DTAP:
boxes
and - checks
Box is an abstraction for everything we want to test - from web server to messages in syslog files. Boxes produces some output to be checked against check rules ( aka checks ). In the previous examples - boxe is just ls command which output redirected to a certain check rule via --box - notation ( meaning read box output from STDIN ). DTAP comes with some predefined boxes user can use them without writing any piece of code, but most of the time boxes - are something a user would write as a chain of Bash commands something likes that:
#!/bin/bash
(
2>&1
cmd1;
cmd2;
cmd3;
# ...
) | tap --box -
Or with a single script:
#!/bin/bash
. /some/box/script.sh | tap --box -
Checks are rules written on formal DSL and executed remotely on DTAP server, so users don't need to install anything, only small tap binary written on golang that interacts with a server send output from boxes to a server and get results back. Checks DSL is based on regular expressions and is super flexible, allowing many things to do including extension by using many general programming languages.
As an example if you look inside path-ok check that verifies file/directory existence you'll see something like this:
generator: <<OK
!bash
echo "regexp: ^^ \"$(config path)\" \$\$"
OK
To list available checks just run:
tap --check_list
Follow double tap web site documentation to get details on using available checks.
It's easy to create new checks and add them to DTAP distribution, if you are instead please let me know. There is quick start introduction into check DSL could be found here
Conclusion.
DTAP is a new kid on the block, that allows to test infrastructure with using just a Bash yet very flexible and powerful. I encourage you to play with it, you can start with installation guide
Thanks for reading
This document (notebook) shows transformation of movie dataset into a form more suitable for making a movie recommender system. (It builds upon Part 1 of the blog posts series.)
The movie data was downloaded from “IMDB Movie Ratings Dataset”. That dataset was chosen because:
The full “Raku for Data Science” showcase is done with three notebooks, [AAn1, AAn2, AAn3]:
Remark: All three notebooks feature the same introduction, setup, and references sections in order to make it easier for readers to browse, access, or reproduce the content.
Remark: The series data files can be found in the folder “Data” of the GitHub repository “RakuForPrediction-blog”, [AAr1].
The notebook series can be used in several ways:
Here are the transformation, data analysis, and machine learning steps taken in the notebook series, [AAn1, AAn2, AAn3]:
Load packages used in the notebook:
use Math::SparseMatrix;
use ML::SparseMatrixRecommender;
use ML::SparseMatrixRecommender::Utilities;
use Statistics::OutlierIdentifiers;
#% javascript
require.config({
paths: {
d3: 'https://d3js.org/d3.v7.min'
}});
require(['d3'], function(d3) {
console.log(d3);
});
#% js
js-d3-list-line-plot(10.rand xx 40, background => 'none', stroke-width => 2)
my $title-color = 'Silver';
my $stroke-color = 'SlateGray';
my $tooltip-color = 'LightBlue';
my $tooltip-background-color = 'none';
my $tick-labels-font-size = 10;
my $tick-labels-color = 'Silver';
my $tick-labels-font-family = 'Helvetica';
my $background = '#1F1F1F';
my $color-scheme = 'schemeTableau10';
my $color-palette = 'Inferno';
my $edge-thickness = 3;
my $vertex-size = 6;
my $mmd-theme = q:to/END/;
%%{
init: {
'theme': 'forest',
'themeVariables': {
'lineColor': 'Ivory'
}
}
}%%
END
my %force = collision => {iterations => 0, radius => 10},link => {distance => 180};
my %force2 = charge => {strength => -30, iterations => 4}, collision => {radius => 50, iterations => 4}, link => {distance => 30};
my %opts = :$background, :$title-color, :$edge-thickness, :$vertex-size;
# {background => #1F1F1F, edge-thickness => 3, title-color => Silver, vertex-size => 6}
Download from GitHub the files:
And unzip them.
Ingest movie data:
my $fileName = $*HOME ~ '/Downloads/movie_data.csv';
my @dsMovieData=data-import($fileName, headers=>'auto');
@dsMovieData .= map({ $_<title_year> = $_<title_year>.Int.Str; $_});
deduce-type(@dsMovieData)
# Vector(Assoc(Atom((Str)), Atom((Str)), 15), 5043)
Here is a sample of the movie data over the columns we most interested in:
#% html
my @movie-columns = <index movie_title title_year genres imdb_score num_voted_users>;
@dsMovieData.pick(4)
==> to-html(field-names => @movie-columns)
| index | movie_title | title_year | genres | imdb_score | num_voted_users |
|---|---|---|---|---|---|
| 3322 | Veronika Decides to Die | 2009 | Drama|Romance | 6.5 | 10100 |
| 1511 | The Maze Runner | 2014 | Action|Mystery|Sci-Fi|Thriller | 6.8 | 310903 |
| 1301 | Big Miracle | 2012 | Biography|Drama|Romance | 6.5 | 15231 |
| 55 | The Good Dinosaur | 2015 | Adventure|Animation|Comedy|Family|Fantasy | 6.8 | 62836 |
Ingest the movie data already transformed in the first notebook, [AAn1]:
my @dsMovieDataLongForm = data-import($*HOME ~ '/Downloads/dsMovieDataLongForm.csv', headers => 'auto');
deduce-type(@dsMovieDataLongForm)
# Vector(Assoc(Atom((Str)), Atom((Str)), 3), 84481)
Data summary:
my @field-names = <Item TagType Tag>;
sink records-summary(@dsMovieDataLongForm, :@field-names)
# +------------------+------------------------+-------------------+
# | Item | TagType | Tag |
# +------------------+------------------------+-------------------+
# | 1387 => 27 | genre => 29008 | Drama => 5188 |
# | 3539 => 27 | actor => 15129 | English => 4704 |
# | 902 => 27 | title => 5043 | USA => 3807 |
# | 2340 => 27 | reviews_count => 5043 | Comedy => 3744 |
# | 839 => 25 | language => 5043 | Thriller => 2822 |
# | 1667 => 25 | country => 5043 | Action => 2306 |
# | 466 => 25 | director => 5043 | Romance => 2214 |
# | (Other) => 84298 | (Other) => 15129 | (Other) => 59696 |
# +------------------+------------------------+-------------------+
One way to investigate (browse) the data is to make a recommender system and explore with it different aspects of the movie dataset like movie profiles and nearest neighbors similarities distribution.
In order to make a more meaningful recommender we put the values of the different numerical variables into “buckets” — i.e. intervals derived corresponding to the values distribution for each variable. The boundaries of the intervals can form a regular grid, correspond to quanitile values, or be specially made. Here we use quantiles:
my @bucketVars = <score votes_count reviews_count>;
my @dsMovieDataLongForm2;
sink for @dsMovieDataLongForm.map(*<TagType>).unique -> $var {
if $var ∈ @bucketVars {
my %bucketizer = ML::SparseMatrixRecommender::Utilities::categorize-to-intervals(@dsMovieDataLongForm.grep(*<TagType> eq $var).map(*<Tag>)».Numeric, probs => (0..6) >>/>> 6, :interval-names):pairs;
@dsMovieDataLongForm2.append(@dsMovieDataLongForm.grep(*<TagType> eq $var).map(*.clone).map({ $_<Tag> = %bucketizer{$_<Tag>}; $_ }))
} else {
@dsMovieDataLongForm2.append(@dsMovieDataLongForm.grep(*<TagType> eq $var))
}
}
sink records-summary(@dsMovieDataLongForm2, :@field-names, :12max-tallies)
# +------------------+------------------------+--------------------+
# | Item | TagType | Tag |
# +------------------+------------------------+--------------------+
# | 902 => 19 | actor => 15129 | English => 4704 |
# | 2340 => 19 | genre => 14504 | USA => 3807 |
# | 1387 => 19 | score => 5043 | Drama => 2594 |
# | 3539 => 19 | country => 5043 | Comedy => 1872 |
# | 152 => 18 | votes_count => 5043 | Thriller => 1411 |
# | 466 => 18 | language => 5043 | Action => 1153 |
# | 1424 => 18 | year => 5043 | Romance => 1107 |
# | 839 => 18 | director => 5043 | Adventure => 923 |
# | 132 => 18 | title => 5043 | 6.1≤v<6.6 => 901 |
# | 113 => 18 | reviews_count => 5043 | 7≤v<7.5 => 891 |
# | 720 => 18 | | Crime => 889 |
# | 1284 => 18 | | 7.5≤v<9.5 => 886 |
# | (Other) => 69757 | | (Other) => 48839 |
# +------------------+------------------------+--------------------+
Here we make a Sparse Matrix Recommender (SMR):
my $smrObj =
ML::SparseMatrixRecommender.new
.create-from-long-form(
@dsMovieDataLongForm2,
item-column-name => 'Item',
tag-type-column-name => 'TagType',
tag-column-name => 'Tag',
:add-tag-types-to-column-names)
.apply-term-weight-functions('IDF', 'None', 'Cosine')
# ML::SparseMatrixRecommender(:matrix-dimensions((5043, 13825)), :density(<23319/23239825>), :tag-types(("reviews_count", "score", "votes_count", "genre", "country", "language", "actor", "director", "title", "year")))
Here are the recommender sub-matrices dimensions (rows and columns):
.say for $smrObj.take-matrices.deepmap(*.dimensions).sort(*.key)
# actor => (5043 6256)
# country => (5043 66)
# director => (5043 2399)
# genre => (5043 26)
# language => (5043 48)
# reviews_count => (5043 7)
# score => (5043 7)
# title => (5043 4917)
# votes_count => (5043 7)
# year => (5043 92)
Note that the sub-matrices of “reviews_count”, “score”, and “votes_count” have small number of columns, corresponding to the number probabilities specified when categorizing to intervals.
my $mat = $smrObj.take-matrices<year>;
my $matUp = Math::SparseMatrix.new(
diagonal => 1/2 xx ($mat.columns-count - 1), k => 1,
row-names => $mat.column-names,
column-names => $mat.column-names
);
my $matDown = $matUp.transpose;
# mat = mat + mat . matDown + mat . matDown
$mat = $mat.add($mat.dot($matUp)).add($mat.dot($matDown));
# Math::SparseMatrix(:specified-elements(14915), :dimensions((5043, 92)), :density(<14915/463956>))
Make a new recommender with the enhanced matrices:
my %matrices = $smrObj.take-matrices;
%matrices<year> = $mat;
my $smrObj2 = ML::SparseMatrixRecommender.new(%matrices)
# ML::SparseMatrixRecommender(:matrix-dimensions((5043, 13825)), :density(<79829/69719475>), :tag-types(("genre", "title", "year", "actor", "director", "votes_count", "reviews_count", "score", "country", "language")))
Example recommendation by profile:
sink $smrObj2
.apply-tag-type-weights({genre => 2})
.recommend-by-profile(<genre:History year:1999>, 12, :!normalize)
.join-across(select-columns(@dsMovieData, @movie-columns), 'index')
.echo-value(as => {to-pretty-table($_, align => 'l', field-names => ['score', |@movie-columns])})
# +----------+-------+------------------------------------------+------------+----------------------------------------------+------------+-----------------+
# | score | index | movie_title | title_year | genres | imdb_score | num_voted_users |
# +----------+-------+------------------------------------------+------------+----------------------------------------------+------------+-----------------+
# | 1.887751 | 553 | Anna and the King | 1999 | Drama|History|Romance | 6.7 | 31080 |
# | 1.817476 | 215 | The 13th Warrior | 1999 | Action|Adventure|History | 6.6 | 101411 |
# | 1.567726 | 1016 | The Messenger: The Story of Joan of Arc | 1999 | Adventure|Biography|Drama|History|War | 6.4 | 55889 |
# | 1.500264 | 2468 | One Man's Hero | 1999 | Action|Drama|History|Romance|War|Western | 6.2 | 899 |
# | 1.487091 | 2308 | Topsy-Turvy | 1999 | Biography|Comedy|Drama|History|Music|Musical | 7.4 | 10037 |
# | 1.479006 | 4006 | La otra conquista | 1998 | Drama|History | 6.8 | 1024 |
# | 1.411933 | 492 | Thirteen Days | 2000 | Drama|History|Thriller | 7.3 | 45231 |
# | 1.312900 | 909 | Beloved | 1998 | Drama|History|Horror | 5.9 | 6082 |
# | 1.237700 | 1931 | Elizabeth | 1998 | Biography|Drama|History | 7.5 | 75973 |
# | 1.168287 | 253 | The Patriot | 2000 | Action|Drama|History|War | 7.1 | 207613 |
# | 1.069476 | 1820 | The Newton Boys | 1998 | Action|Crime|Drama|History|Western | 6.0 | 8309 |
# | 1.000000 | 4767 | America Is Still the Place | 2015 | History | 7.5 | 22 |
# +----------+-------+------------------------------------------+------------+----------------------------------------------+------------+-----------------+
Recommendation by history:
sink $smrObj
.recommend(<2125 2308>, 12, :!normalize, :!remove-history)
.join-across(select-columns(@dsMovieData, @movie-columns), 'index')
.echo-value(as => {to-pretty-table($_, align => 'l', field-names => ['score', |@movie-columns])})
# +-----------+-------+-------------------------+------------+----------------------------------------------+------------+-----------------+
# | score | index | movie_title | title_year | genres | imdb_score | num_voted_users |
# +-----------+-------+-------------------------+------------+----------------------------------------------+------------+-----------------+
# | 12.510011 | 2125 | Molière | 2007 | Comedy|History | 7.3 | 5166 |
# | 12.510011 | 2308 | Topsy-Turvy | 1999 | Biography|Comedy|Drama|History|Music|Musical | 7.4 | 10037 |
# | 8.364831 | 1728 | The Color of Freedom | 2007 | Biography|Drama|History | 7.1 | 10175 |
# | 8.182233 | 1724 | Little Nicholas | 2009 | Comedy|Family | 7.2 | 9214 |
# | 7.753039 | 3619 | Little Voice | 1998 | Comedy|Drama|Music|Romance | 7.0 | 13892 |
# | 7.439471 | 2285 | Mrs Henderson Presents | 2005 | Comedy|Drama|Music|War | 7.1 | 13505 |
# | 7.430299 | 3404 | Made in Dagenham | 2010 | Biography|Comedy|Drama|History | 7.2 | 11158 |
# | 7.270637 | 1799 | A Passage to India | 1984 | Adventure|Drama|History | 7.4 | 12980 |
# | 7.264810 | 3837 | The Names of Love | 2010 | Comedy|Drama|Romance | 7.2 | 6304 |
# | 7.117232 | 4648 | The Hammer | 2007 | Comedy|Romance|Sport | 7.3 | 5489 |
# | 7.046925 | 4871 | Shotgun Stories | 2007 | Drama|Thriller | 7.3 | 7148 |
# | 7.040720 | 3194 | The House of Mirth | 2000 | Drama|Romance | 7.1 | 6377 |
# +-----------+-------+-------------------------+------------+----------------------------------------------+------------+-----------------+
Find movie IDs for a certain criteria (e.g. historic action movies):
my @movieIDs = $smrObj.recommend-by-profile(<genre:Action genre:History>, Inf, :!normalize).take-value.grep(*.value > 1)».key;
deduce-type(@movieIDs)
# Vector(Atom((Str)), 14)
Find the profile of the movie set:
my @profile = |$smrObj.profile(@movieIDs).take-value;
deduce-type(@profile)
# Vector(Pair(Atom((Str)), Atom((Numeric))), 108)
Find the top outliers in that profile:
outlier-identifier(@profile».value, identifier => &top-outliers o &quartile-identifier-parameters)
==> {@profile[$_]}()
==> my @profile2;
deduce-type(@profile2)
# Vector(Pair(Atom((Str)), Atom((Numeric))), 26)
Here is a table of the top outlier profile tags and their scores:
#%html
@profile.head(28)
==> { $_.map({ to-html-table([$_,]) }) }()
==> to-html(:multi-column, :4columns, :html-elements)
| genre:History0.9999999999999999 | language:Mandarin0.3626315299347615 | score:7.5≤v<9.50.2719736474510711 | year:20150.18131576496738075 |
| language:English0.8159209423532133 | reviews_count:0≤v<370.3626315299347615 | votes_count:5≤v<41200.2719736474510711 | year:20140.18131576496738075 |
| genre:Action0.46214109363846967 | score:6.1≤v<6.60.36263152993476144 | title:Hero0.18131576496738075 | country:UK0.18131576496738075 |
| genre:Adventure0.38097093240387203 | country:USA0.36263152993476144 | votes_count:68935≤v<1473170.18131576496738075 | score:7≤v<7.50.18131576496738075 |
| score:6.6≤v<70.3626315299347615 | reviews_count:450≤v<50600.36263152993476144 | reviews_count:37≤v<910.18131576496738075 | votes_count:4120≤v<149850.18131576496738072 |
| country:China0.3626315299347615 | votes_count:147317≤v<16897640.2719736474510711 | year:20020.18131576496738075 | genre:Drama0.1320986315690731 |
| votes_count:14985≤v<343590.3626315299347615 | reviews_count:91≤v<1550.2719736474510711 | director:Yimou Zhang0.18131576496738075 | genre:Romance0.13001981085966202 |
Plot all of profile’s scores and the score outliers:
#%js
js-d3-list-plot(
[|@profile».value.kv.map(-> $x, $y { %(:$x, :$y, group => 'full profile' ) }),
|@profile2».value.kv.map(-> $x, $y { %(:$x, :$y, group => 'outliers' ) })],
:$background,
:300height,
:600width
)
[AA1] Anton Antonov, “Introduction to data wrangling with Raku”, (2021), RakuForPrediction at WordPress.
[AA2] Anton Antonov, “Implementing Machine Learning algorithms in Raku (TRC-2022 talk)”, (2021), RakuForPrediction at WordPress.
[AAn1] Anton Antonov,
“Small movie dataset analysis”,
(2025),
RakuForPrediction-blog at GitHub.
[AAn2] Anton Antonov,
“Small movie dataset recommender”,
(2025),
RakuForPrediction-blog at GitHub.
[AAn3] Anton Antonov,
“Small movie dataset graph”,
(2025),
RakuForPrediction-blog at GitHub.
[AAp1] Anton Antonov, Data::Importers, Raku package, (2024-2025), GitHub/antononcube.
[AAp2] Anton Antonov, Data::Reshapers, Raku package, (2021-2025), GitHub/antononcube.
[AAp3] Anton Antonov, Data::Summarizers, Raku package, (2021-2024), GitHub/antononcube.
[AAp4] Anton Antonov, Graph, Raku package, (2024-2025), GitHub/antononcube.
[AAp5] Anton Antonov, JavaScript::D3, Raku package, (2022-2025), GitHub/antononcube.
[AAp6] Anton Antonov, Jupyter::Chatbook, Raku package, (2023-2025), GitHub/antononcube.
[AAp7] Anton Antonov, Math::SparseMatrix, Raku package, (2024-2025), GitHub/antononcube.
[AAp8] Anton Antonov, ML::AssociationRuleLearning, Raku package, (2022-2024), GitHub/antononcube.
[AAp9] Anton Antonov, ML::SparseMatrixRecommender, Raku package, (2025), GitHub/antononcube.
[AAp10] Anton Antonov, Statistics::OutlierIdentifiers, Raku package, (2022), GitHub/antononcube.
[AAv1] Anton Antonov, “Simplified Machine Learning Workflows Overview (Raku-centric)”, (2022), YouTube/@AAA4prediction.
[AAv2] Anton Antonov, “TRC 2022 Implementation of ML algorithms in Raku”, (2022), YouTube/@AAA4prediction.
[AAv3] Anton Antonov, “Exploratory Data Analysis with Raku”, (2024), YouTube/@AAA4prediction.
[AAv4] Anton Antonov, “Raku RAG demo”, (2024), YouTube/@AAA4prediction.
How time flies. Yet another year has flown by.
Let’s first start with the technical stuff, as nerds do!
Rakudo saw about 1650 commits (MoarVM, NQP, Rakudo, doc) this year, which is about 20% less than 2024. All of these repositories now have “main” as their default branch (rather than “master”).
About 58% of the Rakudo commits were in the development of RakuAST (up from 33% in 2024), of which more than 95% were done by Stefan Seifert. This work, that was supported by a TPRF grant, resulted in being able to build the part of Rakudo that’s written in Raku using RakuAST (often referred to as the “bootstrap“).
However there are still quite a few issues that need to be fixed before RakuAST-based Rakudo can be made the default. And thus be ready for the next language level release. Still, an amazing amount of work done by Stefan Seifert, so kudos!
Geoffrey Broadwell has done a major update of the internal MoarVM Unicode tools. Based on that work Shimmerfairy developed an update of the MoarVM Unicode support from 15.0 to 17.0. This includes support for some new emojis such as FINGERPRINT 
, SPLATTER 
, HARP 
. This means that all of the changes of Unicode 16 and Unicode 17 were implemented to a very high degree. Quite a feat!
Patrick Böker was really busy this year: new script runners not only made execution of CLI-scripts a bit faster, it also made it possible to run CLI-scripts on Windows without any issues.
Again, a lot of work was done on making the Continuous Integration testing produce fewer (and recently hardly any) false positives anymore. Which makes life for core developers a lot easier!
Very important for packagers: Rakudo now again has a reproducible build process, thanks for Timo Paulssen and others.
Also the REPL (Read, Evaluate, Print Loop) has been improved: grammar changes are now persistent, and it’s also possible to enter multi-line comments. Some of these changes were backported from the REPL module, as described in this blog post.
The tests for experimental Raku features (such as :pack, :cached, :macros) have been moved from the Raku test-suite (roast) to the Rakudo repository, as they are technically not part of the definition of the Raku Programming Language.
Sadly, the JVM backend has hardly seen any updates the past year. It was therefore decided to not mention the JVM backend in releases anymore, nor to make sure that there would not be any ecosystem breakage on the JVM backend before a release.
If you want the JVM to remain a viable backend, you are very much invited to get involved!
The most notable new features in the default language level:
NativeCallAfter many years of people asking for this feature, Patrick Böker actually wrote the infrastructure in MoarVM and Rakudo to support calling C-functions that support a variable number of arguments using the va_arg standard.
To give an example, it’s now possible to create a foo subroutine that will call the printf C-function (from the standard library) that takes a format string as the first argument, and a variable number of arguments after that:
use NativeCall;
sub foo(str, **@ --> int32) is native is symbol('printf') {*}
foo "The answer: %d\n", 42; # The answer: 42The printf() C-function is a good example of using varargs.
Writing terminal applications has become much simpler with the support for pseudo-terminals that Patrick Böker has written. This Advent post explains the how and why of this development, and the new Anolis terminal emulator module.
Support is still a bit raw around the edges: pretty sure the coming year will see a lot of smoothing over, so that it can e.g. be used to create a full featured terminal-based Raku debugger!
Hash.new(a => 42, b => 666)A common beginner mistake was trying to create a new Hash (or Map) object by using .new and named arguments. This would silently create an empty Hash (or Map) because by default any unexpected named arguments are ignored in calls to .new. This was changed so that if Hash.new (or Map.new) is called with named arguments only, they will be interpreted as key/value pairs to be put into the Hash (or Map):
dd Hash.new(a => 42, b => 666);
# {:a(42), :b(666)}
dd Map.new(a => 42, b => 666);
# Map.new((:a(42),:b(666)))The Test module now also provides an exit-ok tester subroutine, making it a lot easier to test the exit behaviour of a piece of code. It takes a Callable, and an integer value. It expects the code to execute an exit statement, and will then compare the (implicitly) given exit value with the given integer value.
use Test;
exit-ok { exit 1 }, 1;
# ok 1 - Was the exit code 1?
exit-ok { exit }, 1;
# not ok 2 - Was the exit code 1?
exit-ok { 42 }, 0;
# not ok 3 - Code did not exit, no exit value to checkThe most notable additions to the future language level of the Raku Programming Language:
An implementation of ordered hashes (a hash in which the order of the keys is determined by order of addition) has become available:
use v6.e.PREVIEW;
my %h is Hash::Ordered = "a".."e" Z=> 1..5;
say %h.keys; # [a b c d e]
say %h.values; # (1 2 3 4 5)This will probably get some easier syntactic sugar. Until then, the above syntax can be used.
The following changes are only seen when using RakuAST (by calling raku with the RAKUDO_RAKUAST=1 environment variable set), and thus available by default when the next language level is released.
The RAKU_LANGUAGE_VERSION environment variable can be used to indicate the default language level with which to compile any Raku source code. Note that this does not affect any explicit language versions specified in the code.
$ RAKUDO_RAKUAST=1 RAKU_LANGUAGE_VERSION=6.e.PREVIEW raku -e 'say nano'
1766430145418821670
$ RAKUDO_RAKUAST=1 RAKU_LANGUAGE_VERSION=6.e.PREVIEW raku -e 'use v6.d; say nano'
===SORRY!=== Error while compiling -e
Undeclared routine:
nano used at line 1Although of limited use while RakuAST is not yet the default, it will make it a lot easier to check the behaviour of code at different language levels, e.g. when running tests in the official test-suite (aka roast).
$?SOURCE, $?CHECKSUMThe compile-time variables $?SOURCE and $?CHECKSUM have been added. The $?SOURCE compile-time variable contains the source of the current compilation unit. If for some reason one doesn’t want that to be included in the bytecode, then the RAKUDO_OMIT_SOURCE environment variable can be set.
The $?CHECKSUM compile-time variable contains a SHA1 digest of the source code.
$ RAKUDO_RAKUAST=1 raku -e 'say $?SOURCE'
say $?SOURCE
$ RAKUDO_RAKUAST=1 raku -e 'say $?CHECKSUM'
81892BA38B9BD6930380BD81DB948E4D7A9C14E7
$ RAKUDO_RAKUAST=1 RAKUDO_OMIT_SOURCE=1 raku -e 'say $?SOURCE'
NilThese additions are intended to be used by the MoarVM runtime debugger, as well as by packagers for verification.
Most of the localization work has been removed from the Rakudo core and put into separate Raku-L10N project. And there it gained a few new contributors! For a progress report, checkout out habere-et-dipertire‘s advent blog post titled Hallo, Wêreld!
To make it easier to work with code in a specific localization, each localization comes with a “fun” command line script, and an official one. So for instance, the Dutch localization has a “dutku” (for dutch raku) executable (well, actually a CLI script), but also a “kaas” one. Same for French (“freku” and “brie”), etc. The fun one is usually associated with a favourite foodstuff of the language in question.
$ dutku -e 'zeg "foo"'
foo
$ kaas -e 'zeg "foo"'
foo
$ freku -e 'dis "foo"'
foo
$ brie -e 'dis "foo"'
fooThe RakuDoc v2.0 specification was completed in December 2024, and 2025 was spent implementing it. A compliant renderer is now available by installing the Rakuast::RakuDoc::Render distribution. Work then began on a document management system called Elucid8 which renders the whole Raku documentation (development preview).
From September onwards, Damian Conway and Richard Hainsworth worked on a enumeration system (originally envisioned for RakuDoc v3.0) so that any block – meaning any paragraph, heading, code snippet, formula, etc – can be enumerated simply by prefixing the blocktype with num. It is a far more flexible system than anything encountered in the editor space.
The RakuDoc specification is now at version 2.20.2. The new enumeration specification has not yet been merged to main because work is still being done on getting Rakuast::RakuDoc::Render to implement the new standard.
It really looks like RakuDoc has the potential to becoming the markup language for any type of serious documentation, and a direct competitor to markdown.
The Raku ecosystem has seen quite a lot of developments. Many modules got moved to the Raku Community Modules Adoption Center where they got a new lease on life. But there were also quite a few new modules, and interesting updates to existing modules in 2025.
According to raku.land in 2025, 503 Raku modules have been updated (or first released): up from 367 in 2024 (an increase of 37%). There are now 2431 different modules installable by zef by just mentioning their name. And there are now 13808 different versions of Raku modules available from the Raku Ecosystem Archive, up from 12181 in 2024, which means more than 4.4 module updates / day on average in 2025 (up from 3.9 updates / day).
The modules that yours truly found interesting, so a very personal list! In alphabetical order:
Also in alphabetical order:
A new experimental bot has appeared on the #raku-dev IRC channel: rakkable. It is basically an interactive front end for the new “rakudo-xxx” features of App::Rak. Which in turn is based on the new Ecosystem::Cache module. This allows easy searching in all most current versions of modules in the ecosystem.
For instance: look in the Raku ecosystem for code mentioned in “provides” sections that contain the string “Lock.new” and which also have the string “$!lock”:
<lizmat> rakkable: eco-provides Lock.new --and=$!lock
<rakkable> Running: eco-provides Lock.new --and=$!lock, please be patient!
<rakkable> Found 30 lines in 25 files (24 distributions):
<rakkable> https://gist.github.com/fa2424aebf085ea656b436c63722bf9dThe bot currently only lives on the #raku-dev, but can also be accessed directly without needing the “rakkable:” prefix.
Yes, there’s also non-technical stuff in the Raku world!
The raku.org website has been completely renewed, thanks to Steve Roe who has taken that on. It’s now completely dogfooded: hypered with htmx. Aloft on Åir. Constructed in cro. Written in raku. & Styled by picocss.
The Raku Documentation Project has gained quite a few collaborators, who are working on making the Raku documentation more accessible to new users. One factor making this easier, is that the CI testing for the documentation has become about 4x as fast by using RakuAST RakuDoc parsing.
Yours truly stopped using what is now X (formerly Twitter). It hurt. But Bluesky and Mastodon are good alternatives, and the people important to yours truly have moved to them. If you haven’t yet, you probably should. As well as the people important to you.
Please be sure to mention the #rakulang tag when posting about the Raku Programming Language!
Sadly it has turned out to be impossible to organize a Raku Conference (neither in-person or online) this year. Hoping for better times next year! It just really depends on people wanting to put in the effort!
It was possible to organize a second Raku Core Summit in 2025!
The Rakudo Weekly News has been brought to you by Steve Roe in the 2nd half of 2025 (and for the foreseeable future). With some new features, such as code gists! Kudos!
The Problem Solving repository has seen an influx of 36 new issues the past year. They all deserve your attention and your feedback! Some of them specifically ask for your ideas, such as:
Please, don’t be shy and have your voice heard!
Sadly, Vadim Belman and Stefan Seifert have indicated that they wanted to step down from the Raku Steering Council. They are thanked for all that they have done for Raku, the Raku Community in general, and the Raku Steering Council in particular.
John Haltiwanger has accepted an invitation to join the Raku Steering Council. The seat opened by Stefan Seifert will not be filled for at least the coming 6 months.
After writing a blog post about a Raku Foundation, a problem solving issue and many discussions at the second Raku Core Summit, there finally is a version of the Raku Foundation Documents that everybody can agree on (at least for now).
Yours truly urges the readers to check out The Articles Of Association, as these will be very hard to change once the foundation is established.
If you are really interested in this kind of stuff, please check out the Regulations for the operation of the Raku Foundation as well.
And if you would like to become part of the initial Executive Board or the Supervisory Board, please send an email to [email protected].
Looking back, again an amazing amount of work has been done in 2025! And not only on the technical side of things!
Hopefully you will all be able to enjoy the Holiday Season with sufficient R&R. Especially Kane Valentine (aka kawaii) who is still going strong in their new role:
And on that note: Слава Україні! Героям слава!
The next Raku Advent Blog is only 340 days away!
This document (notebook) describes three ways of making mazes (or labyrinths) using graphs. The first two are based on rectangular grids; the third on a hexagonal grid.
All computational graph features discussed here are provided by “Graph”, [AAp1]. The package used for the construction of the third, hexagonal graph is “Math::Nearest”, [AAp2].
Just see the maze pictures below. (And try to solve the mazes.)
The first maze is made by a simple procedure which is actually some sort of cheating:
The second maze is made “properly”:
The third maze is again made “properly” using the procedure above with two modifications:
Here are the packages loaded for use in the rest of the notebook:
use Graph;
use Graph::Classes;
use Math::DistanceFunctions;
use Math::Nearest;
use Data::TypeSystem;
use Data::Translators;
use Data::Generators;
This sub is used to invoke the Graphviz graph layout engines:
sub dot-svg($input, Str:D :$engine = 'dot', Str:D :$format = 'svg') {
my $temp-file = $*TMPDIR.child("temp-graph.dot");
$temp-file.spurt: $input;
my $svg-output = run($engine, $temp-file, "-T$format", :out).out.slurp-rest;
unlink $temp-file;
return $svg-output;
}
In this section, we create a simple, “cheating” maze.
Remark: The steps are easy to follow, given the procedure outlined in the introduction.
# Maze dimensions
my ($n, $m) = (10, 25);
# Base grid graph
my $g = Graph::Grid.new($n, $m, :!directed);
# Using the edges of the grid graph make a new graph assigned random edge weights
my $gWeighted = Graph.new($g.edges(:dataset).map({ $_<weight> = random-real([10,1000]); $_}))
# Graph(vertexes => 250, edges => 465, directed => False)
Find the spanning tree of the graph:
my $mazePath = $gWeighted.find-spanning-tree
# Graph(vertexes => 250, edges => 249, directed => False)
Shortest path from the first vertex (bottom-left) to the last vertex (top-right):
my @path = $mazePath.find-shortest-path('0_0', "{$n-1}_{$m-1}");
@path.elems
# 56
Graph plot:
#% html
my $simpleMaze = Graph.new($mazePath.edges);
$simpleMaze.vertex-coordinates = $g.vertex-coordinates;
my %opts =
background => "Black",
:!node-labels,
:!edge-lables,
node-shape => 'square',
node-width => 0.7,
node-color => 'SteelBlue',
node-fill-color => 'SteelBlue',
edge-thickness => 52,
edge-color => 'SteelBlue',
size => '10,6!',
engine => 'neato';
$simpleMaze.dot(|%opts):svg
The “maze” above looks like a maze because the vertices and edges are rectangular with matching sizes, and they are thicker than the spaces between them. In other words, we are cheating.
To make that cheating construction clearer, let us plot the shortest path from the bottom left to the top right and color the edges in pink (salmon) and the vertices in red:
#%html
my $gPath = Graph::Path.new(@path);
$simpleMaze.dot(highlight => {Salmon => $gPath.edge-list, Red => $gPath.vertex-list}, |%opts):svg
A proper maze is a maze given with its walls (not with the space between walls).
Remark: For didactical reasons, the maze in this section is small so that the steps—outlined in the introduction—can be easily followed.
Make two regular graphs: one for the maze walls and the other for the maze paths.
# Maze dimensions
my ($n, $m) = (6, 12) »*» 1;
# Walls graph
my $g1 = Graph::Grid.new($n, $m, prefix => 'w');
# Space graph
my $g2 = Graph::Grid.new($n-1, $m-1);
$g2.vertex-coordinates = $g2.vertex-coordinates.map({ $_.key => $_.value >>+>> 0.5 }).Hash;
$g2
# Graph(vertexes => 55, edges => 94, directed => False)
Maze Path Graph:
my $mazePath = Graph.new($g2.edges(:dataset).map({ $_<weight> = random-real([10,1000]); $_}));
$mazePath = $mazePath.find-spanning-tree;
$mazePath.vertex-coordinates = $g2.vertex-coordinates;
$mazePath
# Graph(vertexes => 55, edges => 54, directed => False)
Combined Graph:
my $g3 = Graph.new([|$g1.edges, |$mazePath.edges]);
$g3.vertex-coordinates = [|$g1.vertex-coordinates, |$mazePath.vertex-coordinates].Hash;
$g3
# Graph(vertexes => 127, edges => 180, directed => False)
Plot the combined graph:
#% html
$g3.dot(
highlight => $mazePath,
:node-labels,
background => "Black",
node-width => 0.45,
node-height => 0.2,
edge-width => 4,
size => '10,10!',
engine => 'neato'
):svg
Remove wall edges using a formula:
my $g4 = $g3.clone;
for $mazePath.edges -> $e {
my ($i, $j) = |$e.key.split('_')».Int;
my ($i2, $j2) = |$e.value.split('_')».Int;
if $i2 < $i || $j2 < $j {
($i2, $j2, $i, $j) = ($i, $j, $i2, $j2)
}
# Horizontal
if $i == $i2 && $j < $j2 {
$g4 = $g4.edge-delete( "w{$i2}_{$j2}" => "w{$i2+1}_{$j2}")
}
# Vertical
if $j == $j2 && $i < $i2 {
$g4 = $g4.edge-delete( "w{$i2}_{$j2}" => "w{$i2}_{$j2+1}")
}
}
$g4
# Graph(vertexes => 127, edges => 126, directed => False)
Plot wall graph and maze space graph:
#% html
$g4.dot(
highlight => $mazePath,
:!node-labels,
background => "Black",
node-width => 0.25,
node-height => 0.2,
edge-width => 4,
size => '10,10!',
engine => 'neato'
):svg
Fancier maze presentation with rectangular vertices and edges (with matching sizes):
#% html
my $g5 = $g4.subgraph($g1.vertex-list);
my @path = $mazePath.find-shortest-path('0_0', "{$n-2}_{$m-2}");
say @path.elems;
my @mazeStartEnd = "w0_0", w0_0 => "w0_1", w0_0 => "w1_0", "w{$n-1}_{$m-1}", "w{$n-1}_{$m-1}" => "w{$n-1}_{$m-2}", "w{$n-1}_{$m-1}" => "w{$n-2}_{$m-1}";
$g4.dot(
highlight => {'#1F1F1F' => [|$mazePath.vertex-list, |$mazePath.edge-list, |@mazeStartEnd], Orange => @path},
#highlight => {'#1F1F1F' => @mazeStartEnd},
background => '#1F1F1F',
size => '10,10!',
:!node-labels,
node-shape => 'square',
node-color => 'SteelBlue',
node-fill-color => 'SteelBlue',
node-width => 0.4,
edge-width => 30,
engine => 'neato'
):svg
Let us create another maze based on a hexagonal grid. Here are two grid graphs:
# Maze dimensions
my ($n, $m) = (6, 10) »*» 2;
# Walls graph
my $g1 = Graph::HexagonalGrid.new($n, $m, prefix => 'w');
# Space graph
my $g2 = Graph::TriangularGrid.new($n-1, $m-1);
$g2.vertex-coordinates = $g2.vertex-coordinates.map({ $_.key => $_.value >>+<< [sqrt(3), 1 ] }).Hash;
$g2
Graph(vertexes => 240, edges => 657, directed => False)
#% html
$g1.union($g2).dot(
highlight => $g2,
:!node-labels,
background => "#1F1F1F",
node-width => 0.85,
node-height => 0.85,
node-font-size => 28,
node-shape => 'hexagon',
edge-width => 7,
size => '10,10!',
engine => 'neato'
):svg
Maze Path Graph:
my $mazePath = Graph.new($g2.edges(:dataset).map({ $_<weight> = random-real([10,1000]); $_}));
$mazePath = $mazePath.find-spanning-tree;
$mazePath = Graph.new($mazePath.edges);
$mazePath.vertex-coordinates = $g2.vertex-coordinates;
$mazePath
# Graph(vertexes => 240, edges => 239, directed => False)
Combine the walls-maze and the maze-path graphs (i.e., make a union of them), and plot the resulting graph:
#% html
my $g3 = $g1.union($mazePath);
$g3.dot(
highlight => $mazePath,
:node-labels,
background => "#1F1F1F",
node-width => 0.85,
node-height => 0.85,
node-font-size => 28,
node-shape => 'hexagon',
edge-width => 7,
size => '10,10!',
engine => 'neato'
):svg
Make a nearest neighbor points finder functor:
my &finder = nearest($g1.vertex-coordinates.Array, method => 'KDTree');
Math::Nearest::Finder(Algorithm::KDimensionalTree(points => 544, distance-function => &euclidean-distance, labels => 544))
Take a maze edge and its vertex points:
my $e = $mazePath.edges.head;
my @points = $g2.vertex-coordinates{($e.kv)};
# [[-3.4641016151381225 -2] [-1.7320508075691228 1]]
Find the edge’s midpoint and the nearest wall-graph vertices:
my @m = |((@points.head <<+>> @points.tail) >>/>> 2);
say "Mean edge point: {@m}";
my @vs = |&finder.nearest(@m, 2, prop => <label>).flat;
say "To remove: {@vs}";
# Mean edge point: -2.5980762113536224 -0.5
# To remove: w3 w6
Loop over all maze edges, removing wall-maze edges:
my $g4 = $g1.clone;
for $mazePath.edges -> $e {
my @points = $g2.vertex-coordinates{($e.kv)};
my @m = |((@points.head <<+>> @points.tail) >>/>> 2);
my @vs = |&finder.nearest(@m, 2, prop => <label>).flat;
$g4 = $g4.edge-delete(@vs.head => @vs.tail);
}
$g4
# Graph(vertexes => 544, edges => 544, directed => False)
The start and end points of the maze:
my ($start, $end) = $g4.vertex-list.head, $g4.vertex-list.sort({ $_.substr(1).Int }).tail;
# (w0 w543)
Finding the Maze Solution:
my $solution = Graph::Path.new: $mazePath.find-shortest-path(|$mazePath.vertex-list.sort(*.Int)[0,*-1]);
$solution.vertex-coordinates = $mazePath.vertex-coordinates.grep({$_.key ∈ $solution.vertex-list }).Hash;
$solution
# Graph(vertexes => 50, edges => 49, directed => False)
Plot the maze:
#% html
my @mazeStartEnd = $start, $end, |$g4.neighborhood-graph([$start, $end]).edges;
my $g5 = $g4.union($solution);
my %opts =
:!node-labels,
background => "#1F1F1F",
node-width => 0.8,
node-height => 0.8,
node-shape => 'circle',
edge-width => 40,
size => '10,10!',
engine => 'neato';
$g4.dot(highlight => {'#1F1F1F' => @mazeStartEnd}, |%opts):svg
(Here is the solution of the maze).
[AA1] Anton Antonov, “Day 12 – Graphs in Raku”, (2024), Raku Advent Calendar at WordPress.
[AAp1] Anton Antonov, Graph, Raku package, (2024–2025), GitHub/antononcube.
[AAp2] Anton Antonov, Math::Nearest, Raku package, (2024), GitHub/antononcube.
[AAv1] Anton Antonov, “Graphs” videos playlist, (2024-2025), YouTube/@AAA4prediction.
Hello again! I return during this week of winter solstice to tell you about my experience participating in the Langjam Gamejam. I planned to use Raku, partially so that you could have an advent blogpost to read today, but also because Raku’s builtin support for grammars ensure that I would not get stuck when writing my parser.
I did a few things to ensure that I would be able to complete the game jam. The first was to be realistic about what I could achieve; with seven days of time, I could realistically expect to produce about three weekends worth of working code. This is only going to be a few hundred lines, maybe one or two files total, and prioritizing basic functionality over any fancy features.
I decided to make an idle game.
There’s no rule against deciding that ahead of time. I also decided to use Raku. That’s allowed too. I read the source of a couple different idle games too, mostly to get ideas of what not to do. In particular, big thanks to Idle Game Maker by Orteil, Antimatter Dimensions by Ivar K., and Trimps by the Trimp authors for publishing reasonably-readable source code. All three of these have some notion of programmability, although I ended up doing something quite different.
The first place to start when making a game is with the scenario that the
player will be asked to experience. Games are fundamentally about roleplay,
and roles only exist within scenarios. Fortunately, I didn’t have to come up
with any of this.
Instead, I asked my friend Remi to come up with something interesting. We spent a couple hours adapting an idea for a Zelda-style fishing minigame, with the novel twist that the game would mechanically be an idle game; instead of putting in the athletic effort to retrieve the fish, the player manages a fishing business and delegates the work to employees.
It’s important to allow a genuine brainstorm at the beginning of the process.
Yes-and reasoning is essential for developing concepts. At the same time, it was important that we not plan for work that we couldn’t schedule. Remi committed to drawing a few icons and I committed to carrying out the core of the gamejam objectives:
The first place to start when making a language is with the objects of that language. I don’t mean object-oriented design but the idea that the language expresses some sort of manipulation of reality. What are we manipulating? How do we manipulate them?
At a high level, idle and incremental games are about resource management. They can also include capacity planning. Thus, I decided that resources are the main objects of study. I also needed some way for the player to interact with the game world, and clickable buttons are a traditional way to express idle games. For each button, I’d thought to have a corresponding action in the game, which I’d also express in language.
I also needed to figure out what substrate I’m going to use. I mean, of course I’m using Raku, but how deeply do I want to embed the game? At one end, I could imagine compiling the game into a single blob which runs wholly in the end user’s windowing system or Web browser, so that there’s nothing of Raku in the end product.
At the other end, I could imagine shallowly embedding the game by writing some Raku subroutines and having the game developer write in ordinary Raku. I initially decided to go with the shallowest embedding that would still allow me to use Raku’s syntax for arithmetic: a Raku sublanguage, or Raku slang. Technically, a slang
is its own programming language, or so I was prepared to argue.
One open question concerned the passage of time. A resource evolves in time, perhaps growing or shrinking; it also has some invariants in time, particularly identity. What did I actually want to store internally? All hand-coded games devolve into a soup of objects cross-referenced by string-keyed maps, or at least that was the case for a half-dozen idle games that I’ve looked at. Maybe we can organize all of that into one big string-keyed map?
Another question is how the game will be experienced. I’d assumed that it should be possible to put up some simple HTTP server and run it locally. My notes are unclear, but I think that this is around the first time that I took a serious look at Humming-Bird.
Let’s actually write some code. I started by writing a slang that abused the Raku metamodel. I was inspired by OO::Actors, which introduces an actor keyword, as well as the implementation of the standard grammar keyword. I can just introduce my own resource and action keywords which manage some subexpressions, including Raku arithmetic, and that’ll be my language. To prototype this, I first wrote out a file which I want to be able to load, and then I wrote the parser which handles it.
Here’s a snippet from that first prototype:
resource fishmonger {
flavor-text "employee with knives and a booming voice for telling stories";
eats seafood by 0.017 per second;
eats bux by 15 per second;
converts from seafood into bux by 75 per second;
}
action hire-fishmonger {
flavor-text "employ a fishmonger to sell seafood";
costs 10 bux;
pays 1 fishmonger;
}Several features are very important here. One big deal is that flavor text is inalienable from the resources and actions. I was very conflicted about this. Languages like Idle Game Maker are basically enlightened CSS and HTML; they are extremely concerned with presentation details rather than getting to the essential mechanics and handling time.
At the same time, Remi and I are both big fans of flavor text both for its immersive value as well as for its ability to create a memorable experience. Another important idea here is that costs and pays are two distinct attributes in concrete syntax, even though they’re going to be implemented as the same underlying sort of amount-and-currency pair.
This syntax is a little heavy. I was imagining that this would be a sort of Ruby or Tcl DSL where each command takes a row of arguments, some of which are literal tokens, and imperatively builds the corresponding resources and actions.
At this point, the scenario is complete. The game will have a few natural resources, like plankton, fish, and sharks; a conversion from fish to seafood; employees like fishers, fishmongers, and white mages; and an enhancement that white mages apply to fishers. There is no objective; it’s a population-dynamics sandbox.
After a day of trying to understand Raku’s metamodel, I concluded that a slang is the wrong layer of integration. I really wanted to run an input file through a parser in order to build a small nest of Resource and Action objects in memory, set up an HTTP server displaying them, and repeatedly take one tick per second, integrating changes over all of those objects.
This was a gumption trap for me; I completely lost motivation for a few hours. In those times, it is essential to allow one’s emotions to flow in order to move past them, and also essential to rest in order to restore energy.
After recovering a bit, I cleaned up my repository and thought about what I should do next. I might as well write a proper grammar. What should that language look like? I agonized over this for a few minutes, went through the possibilities of fixity and bracketing, and eventually decided that a nice little S-expression language would work for my needs.
This did mean that I would need to internalize arithmetic, but I also knew one of the standard cheats of game development: it’s okay to not implement arithmetic operations which aren’t actually used. Consider the following snippet:
(resource plankton 1e15
"little crunchy floaty things"
(growth 0.004 /s)
(view water-color (if (< .count 1e20) "clear" "cloudy"))
(view concentration (str (/ .count 2e14)))
)
(action look-at-water-color
"gaze at the ocean"
(enables view plankton water-color)
)
(action measure-plankton
"buy a plankton meter and put it in the water"
(enables view plankton concentration)
(costs 10 bux)
)This is from my prototype. The only arithmetic that’s required is in the views, which format internal numbers about resources into strings. For those, I have a mini-language which allows the user to specify any arithmetic they want, as long as it’s either division or less-than comparisons. The formatting language is strongly typed; the parser won’t allow a non-Boolean operation as an if conditional, for example.
Some other design decisions stand out. Flavor text is now required. Resources have starting counts, which are also required. Rates always end with “/s”, an abbreviation for “per second” that is supposed to easily distinguish them from non-rates.
Gumption management requires not just succeeding, but having a feeling of understanding and competence. I probably could have started on the parser that night, but instead I walked to the bar and speedran Zelda 3, doing any% No Major Glitches and finishing in about two hours and change. Not a superb time, because I grabbed quite a few backups, including an entire extra heart and two bottles; but I didn’t die. The parser can wait until tomorrow.
Parsing an S-expression is really easy, especially when the list of special forms (the words that can legally follow an opening parenthesis) is short. For each special form, we have a rule that parses each of the required components in order, followed by an optional zero-or-more collection of modifiers / attributes / members / components / accessories. The resource special form from Wednesday is parsed with a rule like:
rule thing:sym { '(resource' <s> * ')' }The parser bottoms out on some very simple tokens. For parsing numbers, we parse a subset of what Raku accepts and then use .Rat or .Num methods to convert those strings to live values by reusing Raku’s parser. I may not have been able to reuse arithmetic but I was certainly able to reuse the numerals!
token id { + }
token s { '"' + '"' }
token n { + ('.' +)? ('e' +)? }As I wired up the parser, I also set up a Humming-Bird application. I’m a fan of Ruby’s Sinatra and Python’s Flask, so it seemed like Humming-Bird would be a good fit for me. It doesn’t come with a preferred HTML-emitting library, so I tried a few options. I started with HTML::Tag, which I had added to the project on either Tuesday or Wednesday, but after a few minutes of practical usage, it became totally unusable due to syntactic zones of ceremony (Subramaniam, Seemann, myself): making a fresh HTML tag requires many source characters. I ended up using HTML::Functional, which is much lighter-weight but occasionally allows me to misuse Nil as a string.
I’m hacking out two roles. I’m presenting them here in their final versions; initially they didn’t take any parameters, which was too restrictive. One role is for rendering HTML and the other role is for evolving with each tick.
The %context is all of the resources and actions, and the $resource is the resource currently being acted upon. This sort of late-bound approach is technically too flexible for what I’m building, but I don’t feel like restricting it.
role Render { method html(%context, $resource) { ... } }
role Evolve { method tick(%context, $resource) {;} }I committed, pushed, and asked Remi for feedback.
Remi approves! They’ll make a few cute little icons for some resources. At this point, I stopped and reflected upon what I’d made so far. Pastafarians typically take Fridays off, and I’m not about to work when I could rest instead. What works? What doesn’t work? Where should I spend the rest of my time? What should I have for dinner?
The parser works. The Resource and Action objects operate as nodes of an AST. Exporting the AST as HTML with Render.html() works. Traversing the AST for a tick with Evolve.tick() also works. The Nix environment, which Ihaven’t mentioned yet, also works; I’m using direnv to configure the environment and
install Raku packages.
Arithmetic operations don’t work yet. Actions don’t actually act on anything. Remi’s artwork isn’t visible and I haven’t split out the CSS yet.
I should spend the rest of my time getting the core mechanics of rendering and evolution to work properly. Easy to say; harder to do.
For dinner, I’ll have noodles of some sort, because it’s Friday. I ended up having spaghetti and meatballs in red sauce.
Before dinner, I went to the bar and speedran Mario 1. I played for about 90 minutes but I didn’t finish a single run. On 8-3, the penultimate level, I was repeatedly defeated by a gauntlet of tough enemies.
Humming-Bird got in my way a little; it blocks by default and the
documentation doesn’t explain how to fix it. After reading the relevant code,
I had to change this line:
listen(8080);To have a non-blocking annotation:
listen(8080, :no-block);I also explore how to perform ticks in the background. I do find Supply.interval, but that will let the interpreter exit. Instead, I end up with the following hack:
while 1 { sleep 1; $venture.tick };As I wired up operations and fixing display bugs, I became increasingly stressed as my CSS changes aren’t being applied. By doing some testing, I discovered that the Humming-Bird convenience helper for attaching static directories is not working. I had initially written:
static('/static', 'static');But this doesn’t work, or it had worked on Friday but not on Saturday, or I had somehow mistyped “static”, or any of a dozen other impossible considerations. I knew that I can’t get distracted by this, and I finished up all of the rest of the functionality; the game works, but it’s not styled and Remi’s artwork isn’t visible.
That’s the end of the game jam. I produced a language and a game. However, the game doesn’t display properly and I wouldn’t consider it to be playable. What a frustration.
I’m not done yet! I want to ensure that the release version has Remi’s artwork displayed. First, I hand-wrote the static routes; these do correctly route the CSS and images.
$router.get('/static/style.css', -> $req, $resp {
$resp.file('static/style.css');
});
$router.get('/static/icons/:icon', -> $req, $resp {
my $name = $req.param('icon');
1$name.contains('..') ?? $resp.status(400) !! $resp.file('static/icons/' ~ $name);
});I found a few holes in our scenario. For example, there’s no way to see how many Bux the user has. By default, resource amounts are hidden both to keep the UI uncluttered and to provide a sense of mystery; however, for Bux or seafood, we want to give the user precise numbers. Our existing syntax can fully accommodate this! The view is enabled by an action which doesn’t have any line items (costs or pays) and it prints the .count variable as-is.
(resource bux 1000
"wireless cash"
(view cash-on-hand (str .count))
)
(action check-balance
"become aware of our earnings"
(enables view bux cash-on-hand)
)With these two fixes, we now have a working interface that allows the scenario to be fully accessed! The new view looks like this:
Finally, I’m not going to be able to deploy this version as written. I’ll have to do some reading on production HTTP setups for Raku. When the game loads, it tries to load one image for every visible resource. If more than about five images are requested then the page fails to load. Every action is an HTTP POST which causes everything to reload again.
I imagine that this is properly fixed by adding entity tags to the HTTP backend so that images can be cached. For example, I took the screenshot in the header of this blog post while Firefox was still considering whether it could load the image for plankton; it eventually gives up:
Every load of the page causes a different set of images to not load. This is an unacceptable game experience.
The idea of a declarative idle-game maker is reasonable and it was only a week’s effort to prototype a basic interpreter which simulates a simple scenario. I think that the biggest time sinks were trying to make a slang instead of a deeper language with a parser, fighting with Humming-Bird, and generally trying to keep code clean. On that last point: I found that cleaning up my code was necessary to let bugs and mistakes become visible.
The game comes out to about three hundred lines of Raku and fewer than one hundred lines of S-expressions. It’s within my coding budget for sure. I don’t think that I went for more than I could reasonably accomplish. The entire code is available in this gist. Remi quite reasonably doesn’t want their art uploaded to GitHub, but you can check out more of their work at their website.
Have a happy winter solstice and Holiday!
This document (notebook) discusses number theory properties and relationships of the integer 2026.
The integer 2026 is semiprime and a happy number, with 365 as one of its primitive roots. Although 2026 may not be particularly noteworthy in number theory, this provides a great excuse to create various elaborate visualizations that reveal some interesting aspects of the number.
The computations in this document are done with the Raku package “Math::NumberTheory”, [AAp1].
use Math::NumberTheory;
use Math::NumberTheory::Utilities;
use Data::Importers;
use Data::Translators;
use Data::TypeSystem;
use Graph;
use Graph::Classes;
use JavaScript::D3;
Notebook priming code:
#%javascript
require.config({
paths: {
d3: 'https://d3js.org/d3.v7.min'
}});
require(['d3'], function(d3) {
console.log(d3);
});
First, 2026 is obviously not prime—it is divisible by 2—but dividing it by 2 gives a prime, 1013:
is-prime(2026 / 2)
# True
Hence, 2026 is a semiprime. The integer 1013 is not a Gaussian prime, though:
is-prime(1013, :gaussian-integers)
# False
A happy number is a number for which iteratively summing the squares of its digits eventually reaches 1 (e.g., 13 → 10 → 1).
Here is a check that 2026 is happy:
is-happy-number(2026)
# True
Here is the corresponding trail of digit-square sums:
is-happy-number(2026, :trail).tail.head(*-1).join(' → ')
# 2026 → 44 → 32 → 13 → 10 → 1
Not many years in this century are happy numbers:
(2000...2100).grep({ is-happy-number($_) })
# (2003 2008 2019 2026 2030 2036 2039 2062 2063 2080 2091 2093)
The decomposition of 2026 as 2 * 1013 means the multiplicative group modulo 2026 has primitive roots. A primitive root exists for an integer n if and only if n is 1, 2, 4, p^k, or 2 p^k, where p is an odd prime and k > 0.
We can check additional facts about 2026, such as whether it is “square-free”, among other properties. However, let us compare these with the feature-rich 2025 in the next section.
Here is a side-by-side comparison of key number theory properties for 2025 and 2026.
| Property | 2025 | 2026 | Notes |
|---|---|---|---|
| Prime or Composite | Composite | Composite | Both non-prime. |
| Prime Factorization | 3⁴ × 5² (81 × 25) | 2 × 1013 | 2025 has repeated small primes; 2026 is a semiprime (product of two distinct primes). |
| Number of Divisors | 15 (highly composite for its size) | 4 (1, 2, 1013, 2026) | 2025 has many divisors; 2026 has very few. |
| Perfect Square | Yes (45² = 2025) | No | Major highlight for 2025—rare square year. |
| Sum of Cubes | Yes (1³ + 2³ + … + 9³ = (1 + … + 9)² = 2025) | No | Iconic property for 2025 (Nicomachus’s theorem). |
| Happy Number | No (process leads to cycle including 4) | Yes (repeated squared digit sums reach 1) | Key point for 2026—its main “happy” trait. |
| Harshad Number | Yes (divisible by 9) | No (not divisible by 10) | 2025 qualifies; 2026 does not. |
| Primitive Roots | No | Yes | This is a relatively rare property to have. |
| Other Notable Traits | – (20 + 25)² = 2025 – Sum of first 45 odd numbers – Deficient number – Many pattern-based representations | – Even number – Deficient number – Few special patterns | 2025 is packed with elegant properties; 2026 is more “plain” beyond being happy. |
| Overall “Interest” Level | Highly interesting—celebrated in math communities for squares, cubes, and patterns | Relatively uninteresting—basic semiprime with no standout geometric or sum properties | Reinforces blog’s angle. |
To summarize:
Here is a computationally generated comparison table of most of the properties found in the table above:
#% html
my &divisors-count = { divisors($_).elems };
<is-prime is-composite divisors-count prime-omega euler-phi is-square-free is-happy-number is-harshad-number is-deficient-number primitive-root>.map({ %(sub => $_, '2025' => ::("&$_")(2025), '2026' => ::("&$_")(2026) ) })
==> to-html(field-names => ['sub', '2025', '2026'], align => 'left')
| sub | 2025 | 2026 |
|---|---|---|
| is-prime | False | False |
| is-composite | True | True |
| divisors-count | 15 | 4 |
| prime-omega | 6 | 2 |
| euler-phi | 1080 | 1012 |
| is-square-free | False | True |
| is-happy-number | False | True |
| is-harshad-number | True | False |
| is-deficient-number | True | True |
| primitive-root | (Any) | 3 |
Digits of 2026 represented in the Phi number system:
my @res = phi-number-system(2026);
# [15 13 10 6 -6 -11 -16]
Verification:
@res.map({ ϕ ** $_ }).sum.round(10e-11);
# 2026
Let us create and plot a graph showing the trails of all happy numbers less than or equal to 2026. Below, we identify these numbers and their corresponding trails:
my @ns = 1...2026;
my @trails = @ns.map({ is-happy-number($_):trail }).grep(*.head);
deduce-type(@trails)
# Vector((Any), 302)
Here is the corresponding trails graph, highlighting the primes and happy numbers:
#% html
my @prime-too = @trails.grep(*.head).map(*.tail.head).grep(*.&is-prime);
my @happy-too = @ns.grep(*.&is-harshad-number).grep(*.&is-happy-number);
my %highlight = '#006400' => @prime-too».Str, # Deep Christmas green for primes
'Blue' => [2026.Str, ], # Blue for the special year
'#fbb606ff' => @happy-too».Str; # Darker gold for joyful numbers
my @edges = @trails.map({ $_.tail.head(*-1).rotor(2 => -1).map({ $_.head.Str => $_.tail.Str }) }).flat;
my $gTrails = Graph.new(@edges):!directed;
$gTrails.dot(
engine => 'neato',
graph-size => 12,
vertex-shape => 'ellipse', vertex-height => 0.2, vertex-width => 0.4,
:10vertex-font-size,
vertex-color => '#B41E3A',
vertex-fill-color => '#B41E3A',
arrowsize => 0.6,
edge-color => '#B41E3A',
edge-width => 1.4,
splines => 'curved',
:%highlight
):svg
There is a theorem by Gauss stating that any integer can be represented as a sum of at most three triangular numbers. Instead of programming such an integer decomposition representation in Raku, we can simply use Wolfram|Alpha, [AA1, AA3], or wolframscript to find an “interesting” solution:
#% bash
wolframscript -code 'FindInstance[{2026 == PolygonalNumber[i] + PolygonalNumber[j] + PolygonalNumber[k], i > 10, j > 10, k > 10}, {i, j, k}, Integers]'
# {{i -> 11, j -> 19, k -> 59}}
Here, we verify the result using Raku:
say "Triangular numbers : ", <11 19 59>.&polygonal-number(:3sides);
say "Sum : ", <11 19 59>.&polygonal-number(:3sides).sum;
# Triangular numbers : (66 190 1770)
# Sum : 2026
Here is the number of primitive roots of the multiplication group modulo 2026:
primitive-root-list(2026).elems
# 440
Here are chord plots [AA2, AAp1, AAp2, AAv1] corresponding to a few selected primitive roots:
#% js
my $n = 2026;
<339 365 1529>.map( -> $p {
my &f = -> $x { power-mod($p, $x, $n) => power-mod($p, $x + 1, $n) };
my @segments = circular-chords-tracing($n, with => &f, :d);
@segments .= map({ $_<group> = $_<group>.Str; $_ });
js-d3-list-line-plot(
@segments,
stroke-width => 0.1,
background => '#1F1F1F',
:300width, :300height,
:!axes,
:!legends,
:10margins,
color-scheme => 'Ivory',
#:$title-color, title => $p
)
}).join("\n")
Remark: It is interesting that 365 (the number of days in a common calendar year) is a primitive root of the multiplicative group generated by 2026. Not many years have this property this century; many do not have primitive roots at all.
(2000...2100).hyper(:4degree).grep({ 365 ∈ primitive-root-list($_) })
# (2003 2026 2039 2053 2063 2078 2089)
The number 2026 appears in 18 results of the search “2026 graphs” in «The On-line Encyclopedia of Integer Sequences». Here is the first result (from 2025-12-17): A033483, “Number of disconnected 4-valent (or quartic) graphs with n nodes.” Below, we ingest the internal format of A033483’s page:
my $internal = data-import('https://oeis.org/A033483/internal', 'plaintext');
text-stats($internal)
# (chars => 2928 words => 383 lines => 98)
Here, we verify the title:
with $internal.match(/ '%' N (<-[%]>*)? <?before '%'> /) { $0.Str.trim }
# Number of disconnected 4-valent (or quartic) graphs with n nodes.
Here, we get the corresponding sequence:
my @seq = do with data-import('https://oeis.org/A033483/list', 'plaintext').match(/'[' (.*) ']'/) {
$0.Str.split(',')».trim
}
# [0 0 0 0 0 0 0 0 0 0 1 1 3 8 25 88 378 2026 13351 104595 930586 9124662 96699987 1095469608 13175272208 167460699184 2241578965849 31510542635443 464047929509794 7143991172244290 114749135506381940 1919658575933845129 33393712487076999918 603152722419661386031]
Here we find the position of 2026 in that sequence:
@seq.grep(2026):k
# (17)
Given the title of the sequence and the extracted position of 2026, this means that the number of disconnected 4-regular graphs with 17 vertices is 2026. Let us create a few graphs from that set by using the 5-vertex complete graph (K₅) and circulant graphs.
Here is an example of such a graph:
#% html
reduce(
{ $^a.union($^b) },
[
Graph::Complete.new(5),
Graph::Complete.new(5).index-graph(6),
Graph::Circulant.new(7,[1,2]).index-graph(11)
]
).dot(engine => 'neato'):svg
And here is another one:
#% html
my $g = reduce(
{ $^a.union($^b) },
[
Graph::Complete.new(5).index-graph(13),
Graph::Circulant.new(12, [1, 5]).index-graph(1),
]
);
$g.dot(engine => 'neato', splines => 'curved'):svg
Here, we check that all vertices have degree 4:
$g.vertex-degree.sum / $g.vertex-count
# 4
Remark: Note that although the plots show disjoint graphs, each graph plot represents a single graph object.
Here are a few ways to compute 2026:
sub postfix:<!>(UInt:D $n) { [*] 1..$n }
[
2**11 - 22,
1 + 2 * 3 / 4 / 5 / 6 * 7! * 8 + 9,
1 + 2 + 3 * 4! + 5 / 6! * 7 * 8! - 9,
1 + 2 + 345 * 6 - 7 * 8 + 9,
1 - 2 * 3! * 456 * 7 + 8! + 9,
12 / 3! * 4**5 + 67 - 89,
12**3 + 45 / 6! * 7! - 8 - 9,
123 / 4! * 56 * 7 + 8 + 9,
9! / 8 / 7 / 6 + 5**4 + 321,
9 * 8 - 7 + 654 * 3 - 2 + 1,
987 + 6! + 5 * 4**3 - 2 + 1
]
# [2026 2026 2026 2026 2026 2026 2026 2026 2026 2026 2026]
This section has a few additional (leftover) comments.
GCD and LCD are also extended to work with rationals.integer-digits, is-happy-number, is-harshad-number, is-abundant-number, is-perfect-number, is-deficient-number, abundant-number, deficient-number, and perfect-number.
integer-digits — had lower implementation priority.
#%bash
number-theory is happy number 2026
# True
#%bash
number-theory primitive root list 2026 | grep 365
# 365
[AA1] Anton Antonov, “Numeric properties of 2025”, (2025), RakuForPrediction at WordPress.
[AA2] Anton Antonov, “Primitive roots generation trails”, (2025), MathematicaForPrediction at WordPress.
[AA3] Anton Antonov, “Chatbook New Magic Cells”, (2024), RakuForPrediction at WordPress.
[EW1] Eric W. Weisstein, “Quartic Graph”. From MathWorld–A Wolfram Resource.
[AAn1] Anton Antonov, “Primitive roots generation trails”, (2025), Wolfram Community. STAFFPICKS, April 9, 2025.
[EPn1] Ed Pegg, “Happy 2025 =1³+2³+3³+4³+5³+6³+7³+8³+9³!”, Wolfram Community, STAFFPICKS, December 30, 2024.
[AAp1] Anton Antonov, Math::NumberTheory, Raku package, (2025), GitHub/antononcube.
[AAp2] Anton Antonov,
NumberTheoryUtilities, Wolfram Language paclet, (2025), Wolfram Language Paclet Repository.
[AAp3] Anton Antonov, JavaScript::D3, Raku package, (2021-2025), GitHub/antononcube.
[AAp4] Anton Antonov, Graph, Raku package, (2024-2025), GitHub/antononcube.
[AAv1] Anton Antonov, Number theory neat examples in Raku (Set 3), (2025), YouTube/@AAA4prediction.
This is the 15th of the HARC Stack essays. Previous <=
Don’t forget – HARC Stack combines HTMX with raku Air, Red and Cro to supply a fresh approach to web development.
Two weeks ago, back before HARC Stack became the darling child of https://raku.org fame, we had been Forming. Most of the features of Air::Form were covered in that episode, but Validating was skipped since that is a topic in its own right.
Once again, here’s a top level view of the Air::Form code from Air::Examples.
Let’s zoom in on the form is validated trait:
has Str $.first-names is validated(%va<names>);
has Str $.last-name is validated(%va<name>) is required;
has Str $.email is validated(%va<email>) is required is email;
has Str $.city is validated(%va<text>); The is validated trait comes from the Cro::WebApp::Form class.
As the docs say:
… server-side validation at the field level can be specified by using the is validated($match-me, ‘Message’) trait, which will use the given validation error message if the input values fails to smartmatch against the condition. For example, it could be used with a regex:
has $.username is validated(/^<[A..Za..z0..9]>+$/, 'Only alphanumerics are allowed');
So that’s a regex and an error Str .
Air::Form disables client side (browser) field validation since (i) the overlap with server side validation can be disorienting and (ii) the browser side validation UI is hideous.
If you are familiar with regex (regular expression) syntax, feel free to skip this section.
Here’s a breakdown…
# This regex:
my regex alpha-num { ^<[A..Za..z0..9]>+$ }
# Explanation:
# ^ # anchor: start of string
# <[A..Za..z0..9]> # a character class including:
# # A..Z → uppercase A through Z
# # a..z → lowercase a through z
# # 0..9 → digits 0 through 9
# + # one or more of the above characters
# $ # anchor: end of string
# So it matches a string made only of alphanumeric characters, 1+ long.
say "Hello123" ~~ /<alpha-num>/; # True
say "hi_there" ~~ /<alpha-num>/; # Nil (underscore not allowed)
say "" ~~ /<alpha-num>/; # Nil (empty string not allowed)A regex is a character level language that does pattern matching against strings – either it matches the provided text, or if not rejects it. The ~~ operator is used to apply the regex to a string resulting in a matcher object. For more details dive into the Raku documents here.
Validating form fields ensures that the data users submit is safe, consistent, and free from malicious input. It protects applications from security risks while also enforcing rules like proper formats for emails, numbers, or passwords. Good validation improves user experience by catching errors early and keeping stored data clean and reliable.
The following validators are provided with Air::Form (via the Hash constant %va – error text removed for clarity):
constant %va = (
text => /^ <[A..Za..z0..9\s.,_#-]>+ $/,
name => /^ <[A..Za..z.'-]>+ $/,
names => /^ <[A..Za..z\s.'-]>+ $/,
words => /^ <[A..Za..z\s]>+ $/,
notes => /^ <[A..Za..z0..9\s.,:;_#!?()%$£-]>+ $/,
postcode => /^ <[A..Za..z0..9\s]>+ $/,
url => /^ <[a..z0..9:/.-]>+ $/,
tel => /^ <[0..9+()\s#-]>+ $/,
email => /^ <[a..zA..Z0..9._%+-]>+ '@'
<[a..zA..Z0..9.-]>+ '.'
<[a..zA..Z]> ** 2..6 $/,
...
);It is easy to add your own validators. Just go %va<myval> = /regex/, ‘text’; … and the examples here are easy to copy and extend. More sophisticated validation schemes can be expected in future with the application of raku Grammars.
We are winding up this first series on HARC stack soon … next [& hopefully last] time the topic is Defaulting. Going forward there is more to do – user credentials, blog posts, CRUD and so on.
Thank you for watching this channel. Even better, review the Getting Started advice and try it out for yourself.
As usual, comments and feedback welcome. We are a friendly bunch over on the raku IRC chat and Discord.
~librasteve
This is the 14th of the HARC Stack essays. Previous <=
As if you didn’t know, HARC Stack combines HTMX with raku Air, Red and Cro to supply a fresh approach to web development.
Hot news this week is that the Raku official website https://raku.org has been rewritten entire ly on HARC Stack… read on to learn more about how we eat our own dogfood.
Here is the new site in all its glory…
The site source code is available on Github here https://github.com/Raku/raku.org if you would like to look under the hood.
The Raku community felt that a rewrite of the old site was needed, and were keen to use the various Raku web modules – especially Cro – to implement a new site.
From the HARC Stack side, the feeling was (i) a great opportunity to put the stack to the test at “medium” scale, (ii) a challenge to make a new attractive, informative and effective design and (iii) a great way to show that HARC Stack can serve a real-world website to the Raku community and to the wider world.
HARC Stack is bigly into HTMX and (for now) Pico CSS. These technologies were chosen since they represent a modern approach to website design and deployment. Take a look at their respective websites and you will detect some themes that resonate with the new raku.org site. Simple, clean and informative. Responsive and smartphone oriented.
Choosing these building blocks was partly technical – as covered in previous episodes … HTMX to open the server side to Raku without giving up dynamic user experience and Pico CSS for the semantic HTML and clean, easily maintained source. And partly it was aesthetic, with an eye on ultimately making a web application library that would be a natural choice and lead to a great upgrade to raku.org.
Prior to this project HARC stack had only been put to work for small scale websites – up to 200 unique visitors per day. This fell into line with the notion that HTMX is a better fit for small scale rather than as a replacement to React for the next Google Maps or Facebook (this is a bit of a misnomer, since HTMX has few fundamental constraints to scaling to serve high traffic loads.
HARC stack leans into active code execution to serve HTTP responses – as we have seen here with active content examples such as Counters, Todos and Live Search. This was a deliberate design trade off to maximise the dynamism behind HTMX. [Other choices can be made with the raku Cro module. The Cro template language composes a repository of templates at compile time during site launch and need less CPU cycles to serve static content.]
HARC stack more closely follows the WordPress model where PHP code is run for every page request and a database dip is performed to serve the page. So it was an interesting challenge to stretch raku to a medium scale website such as raku.org.
So prior to the go live, it was important to check the speed of the new raku.org website under moderate load…. here are the results of a distributed test using BlazerMeter to load 20 simultaneous user sessions….
As you can see, the HARC stack response time remains rock solid at around 250-315ms … this is a green on Google Lighthouse – BlazeMeter is a bit more demanding so we get an amber. There are further speed ups we still have in the development backlog such as vendoring the JS libraries, minimising the SCSS and putting all the images locally. Now we have the profiling tools it is realistic to expect for sub 200ms response times from raku HARC stack.
Using basic busy hour calculations, this looks like:
Great — with 20 concurrent users at the peak and an average session duration of 35s, your busy-hour throughput is:
35s=0.009722h
Users/hour at peak = 20/0.009722 ≈ 2,057
To turn that into daily users for a single-peak, typical daily curve, use “Equivalent Busy Hours” (EBH) — the number of peak-equivalent hours your day contains. A typical single-peak consumer pattern is about 6 EBH.
Daily users ≈ Busy-hour users × EBH
Concentrated day (4 EBH): ~8,229 users/day
Typical single-peak (6 EBH): ~12,343 users/day ← likely
Broader peak (8 EBH): ~16,457 users/day
Very broad (10 EBH): ~20,571 users/day
If you don’t have your own hourly distribution yet, using ~6 EBH is a reasonable default → ≈ 12.3k users/day.So much for the theory. Here is how day one of the new raku.org site looks…
Over 1000 visitors per day served without a hitch.
This is only the beginning of the scaling & performance story for raku and HARC stack. We can look forward to a number of dimensions to ongoing improvements:
Thats enough to digest for now … next time we’ll take a closer look at Validating (we inserted this episode since the new raku.org site went live on Saturday!).
Keep watching this channel if it feels good. Even better, follow the Getting Started and try it out for yourself.
As usual, comments and feedback welcome. We are a friendly bunch over on the raku IRC chat and Discord.
~librasteve
The Elucid8 system can be used to create websites based on RakuDoc. The article is to show how to create a minimal website.
Elucid8 ("elucidate") is still being developed, and more information can be found in the Github elucid8-org repositories.
The following need to be present:
zef install Elucid8::Build
zef install Elucid8::Run-locally
Assuming that the zef installs bin/ files to a location in the PATH, then the utilities in the next section should run without problem.
In an empty directory (to be concrete, lets call it webdir), which will be the root for the website build, run the following
elucid8-setupgather-sourceselucid8-buildrun-locallyThen point a browser at localhost:5000
That is all for the minimum site.
Now change the file site-source/en/index.rakudoc.
Then run elucid8-build; run-locally again to see the changes.
Here elucid8-setup copies resources in the Elucid8::Build distribution to create the minimal configuration entries in webdir,
a sample text and some minimal plugins.
The directory structure under webdir will be something like
- config/
- 01-base.raku
- 02-plugins.raku
- 03-plugin-options.raku
- 04-repos
- misc/
- site-sources/
- en/
- index.rakudoc
- examples.rakudoc
Since Elucid8 is being built from the bottom up to be multi-lingual, it is intended that all of these directories can be named in a local variation. The file
config/01-base.rakucontains the tokens that are used within Elucid8::Build.
It is intended that each set of language sources will be in a different repo. gather-sources looks at config/04-repos for files and repo information, clones repositories, runs git blame against them and stores a file description in misc/.
After this step, website will contain the file misc/repo-info.rakuon.
For the minimal website, the RakuDoc v2 specification is pulled from the Raku repository. This document also shows many of the features of RakuDoc.
The config/04-repos shows an example of how to map documents from the repository to subdirectories within publication.
At this step, the build process starts. A processor engine is created that uses the plugins described in config/02-plugins. These are run with options described in config/03-plugin-options.
After this step, misc/ also contains ui-dictionary.rakuon, which will have the English version of the UI tokens. When this dictionary is appropriately edited, other languages will be available.
Elucid8 makes a distinction between the language of the UI and the language of the contents.
A new directory publication/ has now appeared, and this will contain the HTML version of the index file.
At this step, a Cro app is run that takes the HTML in publication and serves the files to localhost:3000.
There are many ways to customise the site:
config/03-plugin-options and the root-domain field of SiteMap, so that the SEO map points to your website.index.rakudoc all the other RakuDoc sources in your local-sources/ directory.Examples of what can be done can be seen in Sandpit repo, for Raku documentation
This is the 13th of the HARC Stack essays. Previous <=
As if you didn’t know, HARC Stack combines HTMX with raku Air, Red and Cro to supply a fresh approach to web development.
Well, Cro has an awesome, declarative way to set up forms in your web application. But… before we get to the HOW, let’s see the WHAT:
Here’s a top level view of the code from Air::Examples. Using the Raku IntelliJ Plugin means I get syntax highlighting and can use with the built in folding editor:
So the form stuff and the whole of the website generation and launch is here. So, once we have investigated the site stuff before (see HARC Stack posts passim), this week the focus is on the new stuff.
Here we use Air::Form for the first time. And we make a simple form – in this case with just 4 inputs – like this:
class Contact does Form {
has Str $.first-names;
has Str $.last-name is required;
has Str $.email is required is email;
has Str $.city;
...
}We declare a new class that does Form …
… consuming the Air::Form role brings in some new declarator traits, is required and is email …
… the sharp-eyed reader will note that I have dropped off the is validated trait for now, more on that below …
… this is what they do:
is required is used to control the <form> <input> required attributeis email is used to set the <form> <input> type to err, emailHere’s what the resulting HTML looks like:
<form name="Contact" hx-post="contact" hx-swap="outerHTML" novalidate="">
<input type="hidden" name="__CSRF_TOKEN" id="__CSRF_TOKEN" value="n0lxuoXN7U2M4DNudzi3ghI8DdJqcOVqFkAlinf0iw2HqgVsK1WmGQ7S1rbCLDTc">
<label for="first-names">First names</label>
<input type="text" name="first-names" id="first-names">
<label for="last-name">Last name</label>
<input type="text" name="last-name" id="last-name">
<label for="email">Email</label>
<input type="email" name="email" id="email">
<label for="city">City</label>
<input type="text" name="city" id="city">
<button type="submit">
Submit Contact
</button>
</form>Air::Form is synthesizing quite a lot of information to cut out repetitive typing for you. For each <input> item, you get:
<label> tag and texttext)Air::Form is creating the overall form tag, and building it as follows:
hx-post target urlnonvalidate (we will validate on the server side)<button>The other part of our class Contact is concerned with the actions on the form, as set out in method form-routes()…
method form-routes {
self.init;
self.submit: -> Contact $form {
if $form.is-valid {
note "Got form data: $form.form-data()";
self.finish: 'Contact info received!'
}
else {
self.retry: $form
}
}
}You need to write this method (or a variant of it) to control the actions on your form.
Essential Methods:
Contact.empty prepares an empty form for the first use in a page (use instead of Contact.new to avoid validation errors – for example where an is required attribute is not yet provided).
Here is an extract from the documents …
Traits are used to describe the kinds of controls that will be used on a form. The full set of HTML5 control types are available. Remember to check browser support for them is sufficient if needing to cater to older browsers. They mostly follow the HTML 5 control names, however in a few cases alternative names are offered for convenience. Taking care to use is email, is url and is telephone is especially helpful for mobile users since these dynamically control the input keyboard layout.
There is no trait for checkboxes; use the Bool type instead.
All of this great stuff is brought into Air::Form from Cro::HTTP::Form.
Thats enough to digest for now … next time we’ll take a closer look at Validating.
Keep watching this channel if it feels good. Even better, follow the Getting Started and try it out for yourself.
As usual, comments and feedback welcome. We are a friendly bunch over on the raku IRC chat and Discord.
~librasteve
This is the 12th of the HARC Stack essays. Previous <=
By the way, HARC Stack combines HTMX with raku Air, Red and Cro to supply a fresh approach to web development.
My favourite HTMX example is the Active Search. Just so cool to be able to do that with server side code – pure genius.
Here’s a top level view of the code. Using the Raku IntelliJ Plugin means I get syntax highlighting and can use with the built in folding editor:
So, we have structured our code into some pieces:
model Person
class SearchBox, class Results, class Searchtable
my $site = ...; $site.serve
model Person
Well, yeah pretty self explanatory. Check out previous editions (e.g. Modelling) and the Air docs for more on how the Red Object Relational Mapper (ORM) works.
class SearchBox
You should be able to recognise the SearchBox on the final webpage movie. It comprises the h3 title and an input field.
Like all three view classes, it does Component to, errr, reflect the fact that it’s a view component with a method HTML {...} that is called when it is loaded.
Note the cool way HTMX applies a live UX indicator on the title when the search is underway.
Be awed by the HTMX combo of hx-put and hx-trigger… which attrs are, of course, defined as Raku Pairs.
:hx-put("$.url-path/search"),
:hx-trigger<keyup changed delay:500ms, search>,So we are sending a put request with the search needle every 500ms during an active search.
class Results
Another small component, this one provides the tbody of the results table.
class SearchTable
This class is the heart of our web application. It has an errr, has-a relationship with an instance of SearchBox and an instance of Results.
The main method HTML here first shows the Searchbox and then an empty table with tbody marked as a target with :id<search-results>.
method search takes the is controller trait specifying PUT as the http-method.
Here’s what the server logs show on startup – illustrating the addition of the search-table//search PUT route by the Cro server…
Air-Examples/bin > ./08-searchtable.raku
theme-color=green
bold-color=red
adding GET search-table/<Mu $id>
adding PUT search-table/<Mu $id>/search
adding GET nav/<Mu $id>
Starting server. Point browser at localhost:3000. Ctrl-C to stop server
[OK] 200 / - ::1
[OK] 200 /css/styles.css - ::1
[OK] 200 /img/bars.svg - ::1
[OK] 200 /img/favicon.ico - ::1
[OK] 200 /search-table/1/search - ::1When this route is hit by the SearchBox, then the method performs the search (against firstName, lastName and email fields; writes the result list back to the $!results instance and then returns the $results instance (which automatically calls method HTML on $!results) and HTMX places the response into the tbody target.
Yikes – it’s dynamic UX magic.
$site = (...)Finally we make an instance, $searchtable, and place it on the index page of our site like this:
This use of Air::Functional should be familiar to all by now!
Last week, I sermonised about Complexity. raku is a large bags of tools for coding in functional, object oriented, declarative or procedural styles. In raku there is no single “way” in which we should write code.
Air::Functional is a great tool for us to code web applications in functional style. Specifically this gives us a clean functional abstraction of HTML tags that can be intertwined with any raku code such as operators, here a for loop which tees up the raku topic so that methods on the topic can take the simple form.firstName and so on.
Functional coding style can be taken much further in raku – there is a hint of this in the .assuming method that helps us to prime a function (sometimes called partial function application).
However, in our goal of taming complexity, OO style also has much to offer. This weeks example leans on raku’s very concise and powerful OO to decompose the inner components of the SearchTable and to make has-a relationships. Most modern coders have OO skills in depth and this is a very natural way to efficiently arrange code to maximise clarity and maintainability.
So raku’s super-power on display here is its multi-paradigm syntax that helps us to bring to bear the best tool for the job.
An hawk eyed reader will have noticed a couple of Raku technicalities used to tune this:
.fc is fold case – both the needle and the haystack are rendered case independenttbody-attrs takes a Hash, so we need to pack the id attr with the %(...) Hash literal$!searchbox.url-path = $.url-path; is passed to the SearchBox at runtime – setting as a default at the attr declaration has SearchBox $.searchbox .= new: :$!title; is too early in the Component construction.serve is a new method on class Air::Base::Site (after Air:ver<0.0.25+>) that greatly simplifies the launch of a HARC stack website on you local machine Keep watching this channel if it feels good. Next time a bit of Forming. Even better, follow the Getting Started and try it out for yourself.
As usual, comments and feedback welcome. We are a friendly bunch over on the raku IRC chat and Discord.
~librasteve
I've recently released a new Raku module called Resource::Wrangler. The idea is to provide a simple way to handle a pretty significant roadblock in the way Raku handles resources.
Raku distributions have a %?RESOURCES variable that stores references to the resources that are declared in the resources object defined in META6.json.
However, complications arise even as early as testing. This is because the tests will be compiled outside of the CompUnit and it's the CompUnit that circumscribes the extent %?RESOURCES.
In other words, different CompUnit, different %?RESOURCES.
But even beyond that, something as seemingly straightforward as having a local copy of the resource to hand to a library can be quite convoluted.
Resource::Wrangler to the rescueBy combining Resource::Wrangler with dependency injection, it is now dead simple to bridge the gap between the distribution's CompUnit, code that makes use of that distribution's CompUnit, and the test files.
Resource::Wrangler utilizes role parameterization, a sort of hyper-flexible Raku-soaked take on implementing "generics" while adding a lot of characteristic spice to the possibility space.
Here is an example directly from the test suite:
class Resourceful {
has $.resources handles <AT-KEY> =
Resource::Wrangler[{ %?RESOURCES }].new;
}
The most sensible default for parameterizing Resource::Wrangler is { %?RESOURCES }, i.e. a block that returns the %?RESOURCES of the current CompUnit.
Resource::Wrangler implements AT-KEY to expose associative indexing to %?RESOURCES, allowing Resourceful to do the same by declaring that $.resources handles <AT-KEY>. This will be demonstrated in the example below.
But then when we go to test, we can set our own Str => IO pairs for Resource::Wrangler to reference:
my %resources = test => "/tmp/test".IO;
my $resourceful = Resourceful.new:
resources => Resource::Wrangler[-> { %resources }].new;
is "/tmp/test".IO, $resourceful<test>, "Dependency injection works as expected";
Note that the block here needs a signature so as not to be mistaken for a single-argument "hash from a hash" initialization by Raku's grammar. Since %?RESOURCES is actually an object rather than a "proper" hash, it doesn't need this special treatment.
I wrote this library to address some obstacles in another project entirely, one which should have significant examples of relatively complex use of Resource::Wrangler.
Until then, thanks for reading.
Sparrow is a Rakulang based framework that allows to automate routine tasks, the beauty of it - one can use prepacked plugins available from public Sparrow repository - https://sparrowhub.io
Sparrow is installed as native Alpine package:
sudo apk add \
--no-cache --wait 120 -u \
--repository=http://dl-cdn.alpinelinux.org/alpine/edge/community \
raku-sparrow6
Set up plugins repository
export SP6_REPO=http://sparrowhub.io/repo
s6 --index-update
Sparrowhub has a lot of plugins, to limit search to Alpine related plugins:
s6 --search alpine
There are now not too many of them for Alpine, eventually I am going to add more, put in comments what sort of plugins one would like to see
To run plugin, use Sparrow cli, you can also pass parameters to a plugin. For example, to validate APKBUILD file one can use apkbuild-strict plugin:
s6 --plg-run apkbuild-strict@path=/path/to/apkbuild/file
That way Sparrow plugins could be run from terminal or get inlined into shell scripts
It'd be great to hear feedback from (Alpine) users, what sort of tasks could be implemented by Sparrow
Kudos to Celeste - Alpine maintainer for raku-sparrow6 Alpine package creation
Recently I've been exploring two programming languages that live well outside the current mainstream but which in past eras of computing have existed much closer -- or directly inside of -- a previous mainstream.
Despite of their decreased visibility today, these languages -- APL and Eiffel -- represent incredible and currently relevant achievements in language design. They also provide unique programming styles in a holistic fashion that my primary personal language Raku does not. 1
APL and Eiffel continue to provide significant value for their users today and thus are also of interest to me by fact that their continued existence is maintained through sheer commercial viability.
Without software companies supporting various commercial implementations, both of these languages would have "died" out years ago. And yet, they did not.
The experience of finding what I consider to be potentially priceless gems just hiding out in plain sight has reminded me of other moments in my life where I have followed my heart and ended up ahead of the curve.
So please, if you will, let me take you on a shopping trip for eyeglasses in the summer of 1999...
I've always been terrible at conforming. Even during my not-so-non-conformist-after-all phase, those early teen years of Hot Topic and JNCO 32" cuff jeans, I was crap at fitting in with the other outcasts.
For me, buying a pair of the thickest black frames I could find in the shop the summer before my second year of high school was the punkest move I could pull at the time. Let yourk inner freak fly regardless of what anyone thinks, that sounds like it should be punk, right?
So, my reasoning went, wouldn't it be punk for me to embrace my dorky side?
It may surprise some readers today, but I was not the only one who thought that these and other "frameless" styles were dope in the late 90s.
This was at the very, very earliest phase of thick (or even plastic of any kind) glasses frames coming back into fashion -- and the word 'hipster' had not yet been revived as a label because the fashion and lifestyle that would become the stereotype was only just beginning. (Crotchety old man voice: "Back in my day, 'hipster' still referred to a subculture from the 50s and 60s who inspired the Beat poets.")
The reason I wanted those glasses: I wanted to physically embrace my decision to go full nerd my sophomore year (I failed spectacularly but that's a story for later).
During a re-watch of Apollo 13 at a farmhouse in Iowa after dropping my sister off for her second year of college, I noticed how dope all the dudes in the control center looked in their solid black frames. I vowed to pick the thickest pair I could find.
Here is a photo from the real command center during Apollo 13. Look at those tough ass frames!
For those that don't remember or weren't there, the glasses aesthetics in the 90s were almost entirely about disappearing the frames. When I showed up on my first day of school as a sophomore, I got reactions above what I was expecting -- and not in a good way.
Special Agent Dana Scully with a "no-nonsense nerd" look that shows off the way the fashionable frame game was done in the 90s.
I never showed up to Henderson with a face tattoo or a mohawk but I think at least in those cases there would have been some degree of fear or respect in the looks of shock and distaste I encountered.
In just a few short years my choice would become the height of fashion but in those first trips through the crowded hallways between classes I can promise you that no one considered these glasses on my face to be fashionable. I remember my punkest friend at the time laughing behind his hand and pointing at me.
"But I don't get it," I would later ask him, "I thought that doing something so non-conformist would be a punk move."
I can't remember what he said, just his head shaking at my naivete.
Then again, hadn't it been me who wanted to put a big flashy nerd signal on my face in the first place?
"What did you expect would happen?", he asked, at some point during that conversation.
I realized I didn't know. I had definitely chosen those glasses as a statement and so reactions from people were to be expected.
I realized I wasn't so shocked or upset about the reactions from the people who hadn't respected me before that day.
It was getting shit on by the so-called non-conformists (and so-called friends, at that) that had been unexpected.
Need proof? If you were able to stand on two legs on US soil at the time, I am betting you can still viscerally remember the year where everyone -- including you -- danced the Macarena non-stop. (No joke, there was no way to escape that year without some person or event forcing you to participate in a round of La Macarena.)
This post is already getting quite long so I will leave a deeper examination of this topic for another time. It may or may not involve juicy corporate failure analysis when it arrives.
Just deciding to want thick black frames in 1999 didn't guarantee that you would have access to them. I visited half a dozen glasses shops before I found that first pair of jet black retro-60s Calvin Kleins.
I wasn't searching out these frames because I wanted to latch onto some upcoming or popular fashion, just like I wasn't following anyone else I knew in high school when I went searching out old classic albums in the local record shop. I'm not saying I was the only one buying records in that school of 1600+ students. But I can say for a fact that I was the only one with thick black rimmed eyeglasses in 1999. I still have the yearbook to prove it.
By the time the term 'hipster' came to life, thick frames were commonplace. This should have great for me because my lenses are so extra that sticking them in thin frames amplifies their Coke bottle nature to deafening levels.
But no, I had to be difficult about it.
Thick rim ubiquity, however, diluted the impact of letting my eyeglasses communicating an inner nerdiness. Eventually those magical Calvin Kleins turned gray and broke but by then they were only a backup pair. The popularity of thick frames led me to go move on to different -- and less attractive on me -- thin frames.
This was a stupid mistake but I promise you it was far from the only one I've made for similar reasons.
It is important to mention that dismissing something as unimportant or trivial or wrong just because it is popular has a long history of negative consequences too.
Just one non-spectacles example: I refused to start a blog in 2005 because I considered it to be a saturated space full of vanity projects and definitely too mainstream for me to be able to make an impact or "able to speak my real truth". (What an idiot! - ed.)
Actually, I have. I just maybe haven't yet phrased it in it's most common manifestation: be true to yourself.
I liked records and discovering lost classics (and trash rock, so much dollar bin trash rock), so I bought records. I wanted to look like a nerd from NASA circa 1969, so I looked for the right pair of frames until I found them.
These are far from my only "early arrivals" in terms of fashion, book series, technologies, and the like. I land on these latent-but-soon-to-be-explosive gems for the simple reason that I could care less about whether they are currently perceived as cool.
I've been alive long enough to know that no new solution arrives without delivering the next generation of problems. Your hot framework or language today is going to peak and ebb away as a new generation rebels against your dominance and against the inevitable gaps and flaws in your generation's solutions.
It is highly unlikely that they will either be charitable in their critique or particularly informed about the historical dynamics that led to the constraints that they are now rebelling against. They will almost certainly throw the baby out with the bathwater, leaving themselves exposed to the flaws that will act as seeds of their own cyclical downfall.
What is new, what is cool, these are so transient as to be irrelevant.
If you get a job with a current hot tech stack and you actually enjoy that job enough to remain for five or ten years, when you decide to depart that hot stack will almost certainly be considered a hot pile of crap by a non-trivial segment of "cutting edge" programmers.
Don't take it personally. It is in fact a required aspect of marketing this new current hot tech stack that they paint your previously-viewed-as-revolutionary technologies as untrustworthy garbage.
This is what the old sage types mean when they nod knowingly and say "the blade's edge cuts both ways."
It has taken me a long time to come to a place where I accept myself as capable of making a living on my own, pursuing only my own interests towards only my own ends.
It's an important realization precisely because the sense of all-encompassing despair that I fell into on my way into a multi-year burnout, that despair came from a place of irreconcilable differences between what makes me okay with life and fulfilled as a human being and what a day in a huge corporate office looks like (hint: Hell has an open floor plan).
So, do I want to "climb the ladder" and join some other enormous corporation to spend the major portion of my functionally useful life for the enrichment of a large pyramid of stakeholders that could care less about workers?
Amazingly, as little as these enormous companies seem to ultimately care about their employees (at least relative to the Holier-Than-Life-Itself Bottom Line), it often seems that they care even less about the products they put those workers to use towards.
And I'm supposed to live my life improving software objects, from functions to modules to entire systems, that will never receive the love and attention that they deserve?
A waking life of always either creating the next so-called "Minimum Viable Product" (deceivingly abbreviated as an "MVP"), or working on the un-loved guts of an MVP that was only intended as a quickly-replaced "Proof of Concept" that instead became an unchangeable bedrock of technical debt? -- this being the natural result of impatiently waiting business cases flocking to the "POC" "MVP" and essentially showing management's bluff that anything besides the most bare minimum viability of a product was going to land in production?
No. Thanks for your kind offer, but no.
This moment of self-realization -- of realizing what I don't want -- is usually the first stage of a sequence of events that leads me to a new cool thing before the hype cycle arrives.
I don't believe in "discoveries" or "discoverers" so that's never been what I'm about. I sometimes remark on how I "did it before it was cool" but rarely in an unironic way (this essay outweighs any previous unironic expression by a long mile).
Put another way, I could care less about planting my flag somewhere unless that's a place I actually give a shit about being.
To that end, the programming languages that I will be blogging about are ones that I am using (Raku) and learning about (APL and Eiffel) with a great deal of thought and intention -- and with absolutely zero regard for what's fashionable in software development.
Instead I'm focusing on the many ways that our current approaches are based on largely unconscious choices that are steeped in would-be-comic-if-not-so-tragic cargo cult practices -- choices are made today within a narrow focus and to resemble the choices we made yesterday but this time we will try to do it without the choices we didn't like.
Will it actually solve the issue? Well, the fact that we already have good cause -- thanks to accumulated technical debt -- to patch the whole thing again tomorrow should tell us all we need to know on that front. (Code refactoring targeted at alleviating technical debt is a pinnacle form of crafting the structures of tomorrow to look and act exactly like the structures of yesterday, minus whatever annoyances that we can afford to fix at the moment).
Moments of self-reflection don't need to be rare. Right now I'm only even interested in a continued career in programming if it involves using systems that have a track record of not even being capable of encountering entire categories of problems that I've been facing with mainstream solutions. (These problems can be cultural as well as technical and I look forward to writing more about this topic soon.)
Only time will tell how well I have decided... but I have lived through several reasons to embrace and trust my instincts.
With any luck, this will be another wave of a lifetime.
First-class array programming for APL and design by contract for Eiffel. Raku, the polyglot of programming styles, has plenty of syntax and semantics that are applicable and amenable to these styles -- but that polyglot nature keeps Raku from centralizing them the way they are in APL and Eiffel. I am not saying this is a mark against Raku._↩
In a previous article, I described how to take a web page source file, edit it, then send the edited source back to Github.
In that post, the authorisation token id-token was one I generated for myself as a Github user. It would also be possible to assign to the hallowed glade (see previous article) an authorisation for a pseudo user or bot specifically set up to take user generated editing.
However, when the edit is merged into the source, the author of the commit would be the bot, and not the author of the edit.
Suppose, though, we want a 'credits' page listing authors and the number of commits they have made. We could run a command in the repo such as:
git log --pretty="%an %n" | sort | uniq -c | sort -n -r
and filter the results into a table. But authors who have entered our hallowed glade will not be listed.
Another concern is spamming. If we require a community user to have a Github identity, then Github will handle authorisation. If there is spamming by a forest troll, it is in Github's best interest to hold that troll to account.
This post explains (mostly to myself) the steps that are needed to insert a login layer.
When I finally got the whole thing working, it was like a weird dance between three different entities. Looking at the documentation for the first time, all the steps seemed odd and unrelated, but once they were all put into motion, there is a logic to the whole thing.
We already have a hallowed glade, which is the suggestion_box server. The server runs in a docker container, and has a Cro server to handle input from a websocket, a Cro client to interact with Github to create a new branch, store the edited file, and raise a Pull Request.
Github allows the owner of a repo to register a 'Github app' and assign it permissions. So, the token granted during the authorisation process is a combination of the permissions of the app and those of a user.
It is always scary for a user to have to authorise someone else to do something on their behalf, so here is what Github says about the authorisation token:
A token has the same capabilities to access resources and perform actions on those resources that the owner of the token has, and is further limited by any scopes or permissions granted to the token. A token cannot grant additional access capabilities to a user. Github authorisation documentation
Here is diagram of the interaction between the browser, the server (in our case suggestion_box) and Github from a great article by Tony on OAuth2:
Suppose we add in the request that if an editor wants to make edits on several documents (which means the page is refreshed), then it is inefficient for Github to generate an id-token for each page.
We can keep an object in local storage with the name of the editor and the time the first submission is accepted.
Note that Github issues its web tokens for a fixed period of time, by default 8 hours.
A pre-requisite is that the Github app, which is our suggestion-box is already registered with Github.
The brief laid out above suggests the following series of steps:
editor field in the submission form must be consistent with Github rules:
-, with a minimum of three characters and a maximum of 39. -.state string that contains information for the suggestion_box to be able to issue a successful submission message back to the browser.Inside the suggestion_box server:
In this scheme,
In the previous post, the Cro app had a route for the websocket and a Client section to handle putting suggestions into Github.
We need to modify the Client section to use the editor's id-token, but essentially it remains the same.
We also need to add a route which is used by Github to send login information to the server. In addition, there needs to be a way for the webserver route and the authorisation route to interact.
These modifications imply a shared resource to match editor-name, id-token, id-token expiration date. Since Cro assumes concurrency, the storage has to be thread-safe and ensure that only one thread at a time can access it.
Although the Cro documentation uses
OO::Monitorfor this purpose, I prefer the simplerMethod::Protectedmodule. Theis protectedtrait ensures that only one thread at a time can access the shared resource. For example,
use Method::Protected;
class Editor-Store {
#| has key = editor, with two attributes :token and :time (a Date::Time)
has %!storage;
#| if False, makes sure editor key is deleted
method is-editor-active( $editor --> Bool) is protected {
return False unless %!storage{ $editor }:exists;
return True if %!storage{ $editor }<expiration> > (now.DateTime + Duration.new(10 * 60));
sink %!storage{$editor}:delete;
return False
}
#| if there is no editor, an emptry string is returned
method get-token( $editor --> Str) is protected {
if self.is-editor-active( $editor ) { %!storage{$editor}<token> }
else { '' }
}
#| expiration date
method expiration( $editor ) is protected { %!storage{$editor} }
#| returns the remaining time in seconds as an integer
method time-remaining( $editor --> Int ) is protected {
if self.is-editor-active( $editor ) { ( %!storage{ $editor }<expiration> - now.DateTime).Int }
else { 0 }
}
#| adds editor
method add-editor( Str $editor, DateTime $expiration, Str $token ) is protected {
%!storage{$editor} = :$expiration, :$token;
}
}
The Websocket route of the Cro app needed refactoring completely. Instead of getting all the information for an edit from the browser, typically, the edit suggestion comes first but the editor and their id-token comes later. So the Webocket needs to create a promise and also to put a timer on it.
My first-draft solution is (the code needs to be cleaned up somewhat):
get -> 'suggestion_box' {
web-socket :json, -> $incoming {
supply whenever $incoming -> $message {
my $json = await $message.body;
# first filter out the handshake signal for opening a websocket
if $json<loaded> {
say strftime(DateTime.now, '%v %R') ~ ': connection made'
if $debug;
emit({ :connection<Confirmed> })
}
else {
if $debug {
say strftime(DateTime.now, '%v %R') ~ ': got suggestion, now at ' ~ +@suggestions;
for $json.kv -> $k, $v {
say "KEY $k =>\n$v"
}
say "edit suggestion finished\n";
}
my $response = sanitise( $json ); # sanitise returns an error message or 'ok'
my $editor := $json<editor>;
if $response ne 'OK' {
emit( {
:timestamp( DateTime.now.Str ),
:$response,
:$editor,
})
}
# handle the socket with an active editor
elsif $store.is-editor-active($editor) {
say strftime(DateTime.now, '%v %R') ~ ': editor is registered' if $debug;
# too little time left for token
if $store.time-remaining( $editor ) <= 10 * 60 {
say strftime(DateTime.now, '%v %R') ~ ': not enough time' if $debug;
emit( {
:timestamp( DateTime.now.Str ),
:response<TooLittleTimeOnToken>,
:$editor,
})
}
else {
say strftime(DateTime.now, '%v %R') ~ ': handling with stored token' if $debug;
$json<id-token> = $store.get-token($editor);
@suggestions.push: $json;
emit( {
:timestamp( DateTime.now.Str),
:response<OK>,
:$editor,
:expiration( strftime($store.expiration($editor), '%v %R'))
} )
}
}
# the editor does not have a token, but may have in some period of time
else {
say strftime(DateTime.now, '%v %R') ~ ': editor without authorisation' if $debug;
my $timestamp;
my $response = 'NoAuthorisation';
my $expiration = '';
my $token = '';
my Promise $tapped-out .= new;
my $tap = $see-new-auths.tap( -> %a {
if %a<editor> eq $editor {
$timestamp = DateTime.now.Str;
$response = 'OK';
$token = %a<token>;
$expiration = %a<expiration>;
$tapped-out.keep;
}
});
await Promise.anyof(
$tapped-out,
my $timer = Promise.in($auth-wait-time).then: {
$response = 'NoAuthorisation';
}
).then( { $tap.close } );
if $response eq 'OK' {
$json<id-token> = $token;
@suggestions.push: $json;
say strftime(DateTime.now, '%v %R') ~ ": authorising suggestion {$json.raku}, now at " ~ +@suggestions if $debug;
}
emit( %( :$timestamp, :$response, :$expiration, :$editor) )
}
}
}
}
The $tapped-out Promise is created so that it can be kept with .keep inside the code that listens for an authorisation event.
Then a separate Promise composed of the $tapped-out Promise and a timer Promise is created so that whichever comes first triggers the next step. If the timer exits before an authorisation, then the edit suggestion is discarded, otherwise it is combined with the id-token and queued.
In the code section above, the webserver is a supply and when the emit sub is called, the hash argument is mapped by Cro into a JSON object and returned to the browser that has connected to the Cro app. So it can be picked by the Javascript's websocket's onmessage function, and the data used by the browser program.
When a Github app is registered, a route is supplied for the authorisation data. Consequently, the server section of the Cro app has to be set up to service this route. I chose the route /raku-auth (and in my code comes before the websocket, but since we are dealing with concurrent processes, the order of websocket and raku-auth is irrelevant):
get -> 'raku-auth', :%params {
CATCH {
default {
content 'text/html', '<h1>Raku documentation</h1><p>Authorisation error.</p><p>Please report</p>';
say 'error is: ', .message;
for .backtrace.reverse {
next if .file.starts-with('SETTING::');
next unless .subname;
say " in block { .subname } at { .file } line { .line }";
last if .file.starts-with('NQP::')
}
}
}
my %decoded = from-json( base64-decode( %params<state>).decode );
say strftime(DateTime.now, '%v %R') ~ ': got from Github params: ', %params , ' state decoded: ', %decoded
if $debug;
my $editor = %decoded<editor>;
my $resp = await Cro::HTTP::Client.post(
"https://github.com/login/oauth/access_token",
query => %(
:$client_id,
:$client_secret,
:code( %params<code> ),
),
);
# Github returns an object with keys access_token, expires_in (& others not needed)
my $body = await $resp.body;
my %data = $body.decode.split('&').map(|*.split("=",2));
# first store the data for future suggestions
my $token = %data<access_token>;
my $expiration = now.DateTime + Duration.new( %data<expires_in>);
$store.add-editor($editor, $expiration, $token );
# next put the data in a stream for suggestions that have already arrived
$auth-stream.emit( %( :$editor, :$expiration, :$token ) );
content 'text/html', '<h1>Raku documentation</h1><p>Editing has been authorised.</p><p>Thank you</p>';
}
get -> 'suggestion_box' {
The CATCH phaser is only ever triggered if there is an error processing the route. If it is triggered, then the string after content is sent back to Github, which displays it in the tab containing the authorisation button. The HTML could be improved.
At the end of the code, the content sub similarly sends back an HTML string to indicated authorisation is successful.
When a route has a query appended to it, the Cro route function get captures the data into a named hash, which I have called %params. Github recommends that a state variable is sent when sending the user to Github for authorisation. I have chosen to send the name submitted by the editor with Base64 encoding. Since the editor name in the form and the user's Github id could be different, this adds some complexity to improve security.
I have to say that the Cro syntax is easy to work with. When the data is transmitted as a JSON, there is the named :body field, and with the data is transmitted as a query, there is the named :query field. Otherwise the get sub has a consistent syntax. Compare this to a cURL command.
The parameters sent to the route include a code field, which is then returned (again as a query) to Github. When it receives the code, it returns the id-token for the editor. This two-fold handshake makes it difficult to impersonate someone else.
In addition, Github returns the number of seconds the id-token is valid for. So this needs to be combined into a DateTime both for the suggestion-box server and the browser.
Finally, the editor name, id-token and expiration date are both stored in a thread-safe Hash, and placed in an event stream into which the websocket has tapped.
Raku's concurent structures make it easy to set up the event loop into which the raku-auth route injects information and the websocket listens for information. At the start of the code we have
my $auth-stream = Supplier.new;
my $see-new-auths = $auth-stream.Supply;
As can be seen from the code fragments, the raku-auth route has the line $auth-stream.emit( %(...) ) and the websocket code has the line my $tap = $see-new-auths.tap( -> %a { ... }).
The first stanza supplies an item, in this case as hash, while the second listens for all items supplied. The second then determines which item to react to.
One of the pieces of data required by a Github app is a 'secret' for the app. There are also several items to be provided to the Cro app.
Since this Cro app is intended for a docker container, the configuration data is conveyed in Environment variables. So at the start of the app, several secret variables are extracted as (just an example here):
my $client_id = %*ENV<CLIENT_ID>;
and these data can be temporarily saved in an environment file env-file. Then when the docker image is invoked the data can be supplied:
sudo docker run -d --rm --env-file env-file my-docker-image
As can be seen from the diagram above, getting authorisation is delicate dance, and it took me days to stop the browser, Github and the suggestion-box treading on each others' toes.
First, the interaction between Github and the suggestion-box server to exchange a code for an id-token are all conducted using the query format. That is a url ending ?data-item=stuff&item-two=nonsense, which is in turn Base64 encoded. All the Github API calls use JSON data with authorisation headers.
In hindsight, we can recognise that the OAuth protocol is an industry standard, while the Github API protocols are not, so can be different. But it took me a while to work out what the strange data was being received by the server. I did not come across this behaviour as being explicitly documented.
Second, Github requires that the suggestion-box server is registered by the owner of the repo as a Github app and specific permissions need to be allocated to it. There are dozens of permissions in over a dozen categories and the correct ones have to be given to the Github app. I thought - mistakenly - that the permissions for Pull requests was what I needed.
Actually this was the last bug I had to overcome and it took a couple of days to figure out that the error was not in the server code, but that I had not allocated enough permissions.
To summarise, the suggestion-box server uses four separate Github API calls and the permissions are different:
api.github.com/repos/{$repo-name}/git/ref/heads/main to get the commit sha for the repo-name - this is public data and the permission is mandatory for an App, so it does not need to be specifically addedapi.github.com/repos/{$repo-name}/git/refs (with data) to create a reference - this requires the Contents permission with read/writeapi.github.com/repos/{$repo-name}/contents/{$file-path} to supply the edited - this requires the Contents permissionapi.github.com/repos/{$repo-name}/pulls to create a pull request of the new branch - this requires the Pull Request permissionSome 'simple' requests have complex solutions. Although Cro and Raku's concurrent structures take a while to understand, they are easy to apply.
The ask, expressed by our website users, is simple enough, edit a source from a GitHub repo in a browser, then save the edited version back.
How difficult is that? It's easy to point the browser at the file in the repo, and GitHub has an editor ...
The problem comes from authorisation, and bad actors wanting to add stupid stuff to files - like: I'm the brilliantish shop for high-ent wazzoo's. Come take a look at http://rip-me-off.darkweb.cosmo
Github's editor is easy to use - if you have commit permissions on the repo, otherwise:
But its only one letter!??
Also, our website text source is written in RakuDoc (not MarkDown, but that's another story) and so it would be nice to be able to see what the change looks like. As always a seemingly trivial mistake, such as deleting a < in the wrong place, can have severe effects.
So, it would be nice to see an HTML rendering of the edited text before submitting it back.
It seems doable:
fetch function. Rakuast::RakuDoc::Render into a docker container, wrote a very simple Cro app that has all the logic for a websocket. The app expects RakuDoc source, and sends back the rendered HTML, which is then added into a div next to the online editor. I'll not explain tasks 1-3 above and focus on task 4.
While getting the content is easy, returning it is hard. The API documentation is written for developers who understand Github and HTTP requests.
I spent two days wandering though the API documentation, searching for tutorials, (and Github's own discussion forum seems to be flooded with trolls, so is not a good place to look).
I even resorted - gasp - to ChatGPT. Whilst the code it suggested was not remotely what I wanted, the 'solutions' demonstrated a remarkably simple truth. Perhaps I should have realised it on my own, but in all my documentation searches I had not read a hint about it.
The simple truth: getting a patch of a file to the repo maintainers could only be done in a sequence of steps, not the single step that I was looking for.
It is obvious that Github wants to ensure that only authorised users can change the state of a repo. Authorisation is accomplished by accompanying any request for a change with a token that has a limited duration, is unique to a recognised user on Github, and has permissions for the repo.
While it is possible to get and use such a token for an arbitrary user, by sending them to Github to log in, the process is an added layer of complexity. For simplicity, we shall consider that a token has been generated and call it id-token. (For development, I used one generated for myself)
In generalised terms, here are the steps to sending a suggestion for editing a file to the Github repo. (I'll explain some of the jargon with each step).
Obtain the contents of the file and its sha from repo.
Raku/doc Github storage, where Raku is the organisation that owns the storage, and doc is the repo in which Raku holds its documentation suite.sha is the encrypted sum of the entire file, and is presented as a 40 digit hexadecimal string. The maths of shas is quite interesting but not relevant here. Suffice it to say that if a file is changed at all, the shas of the two files (before the edit and after it) will be different (for the pedantic, I'll add 'with high probability').Obtain the latest sha or commit for the repo.
Create a new branch or reference point in the repo.
Move a copy in BASE64 format of the edited file to the new branch
Raise a PR for the branch
PR is a pull request, and it is a suggestion to the repo maintainer to merge the suggested changes of the file into the main content of the repo.Even though only one letter in one file may need to be changed, the API is set up (reasonably) for many changes to many files.
Can we do this in the browser? Uh, well ... yes, but No.
The problem is that everything in a browser can be examined by the world. So if you put an id-token in a browser, it can be extracted from the browser, and then a bad actor can use the token to do silly stuff.
So, each of the steps above which require an id-token have to be executed from a place that can be defended, such as inside a container where the id-token can be hidden and renewed on a regular basis.
The solution is to create a safe location - hallowed glade - in which the id-token can be stored, and from which the calls to Github can safely be executed.
So let us create a container with a Cro application. The container will have a websocket to accept the edited file from the browser, and then separately interact with Github.
The Cro documentation was written by a genius for developers with experience, unlike me. They are daunting and confusing - it took days of work to figure things out. However, Cro does make it easy to set up the steps listed above.
The first thing to realise is that Cro::HTTP::Client and Cro::HTTP::Server / Cro::HTTP::Router are - at least externally - independent of each other, but we will need both.
We will need the Server to listen for the data coming down the websocket, and then Client to move the edited file to the Github repo.
Since step 1 above is done in-browser, it needs to be implemented in Javascript. I will not deal with that here, except to say, that when the content is fetched from Github, the sha of the file is saved. The in-browser code finally sends along a websocket a JSON object with the following fields:
The last three are not strictly needed, but can be used for some sanity testing.
Lets assume the web socket has a route suggestion_box, then the following is the code for a Cro application to get the data
use Cro::HTTP::Log::File;
use Cro::HTTP::Server;
use Cro::HTTP::Router;
use Cro::HTTP::Client;
use Cro::HTTP::Router::WebSocket;
use DateTime::strftime; # a helper to provide nice time formats
my $patch-limit = 5 * 2 ** 10; # 5k limit of chars
my $comment-limit = 1 * 2 ** 10; # 1k limit of chars
my $pr-update-duration = 10 * 60 * 60; # every ten minutes
my $host = '0.0.0.0';
my $port = 60005;
my @suggestions;
my $busy = False;
my Cro::Service $http = Cro::HTTP::Server.new(
http => <1.1>,
:$host,
:$port,
application => routes(),
after => [
Cro::HTTP::Log::File.new(logs => $*OUT, errors => $*ERR)
]
);
say strftime(DateTime.now, '%v %R') ~ ': starting up.';
$http.start; # this starts the server part of the application
# the following is an asynchronous structure which listens for events
# then does the block for that event. The main event is to stop
# the app when a control signal is raised.
react {
whenever signal(SIGINT) {
$http.stop;
say strftime(DateTime.now, '%v %R') ~ ': closing down.';
done;
}
whenever Supply.interval($pr-update-duration) {
# every pr-update-duration seconds, this block is run
# it calls the code for raising a PR
unless $busy {
$busy = True;
raise-pr( @suggestions.pop ) while @suggestions;
$busy = False;
}
}
}
# these routes are what define the application
sub routes() {
route {
get -> 'suggestion_box' { # the route for the websocket
web-socket :json, -> $incoming {
supply whenever $incoming -> $message {
my %json = await $message.body;
if %json<sha> { # no sha, no play
my $response = sanitise( $json );
# place limitations on the incoming data
# add 'cancel' to ignore data
@suggestions.push: $json.hash.clone
unless $json<cancel>;
# store the data to be sent as PR
emit({
:timestamp( strftime(DateTime.now, '%v %R')),
:$response
})
# send back to the websocket JSON with
# a time stamp and a reponse code
}
elsif $json<loaded> {
# send back a handshake when websocket opens
emit({ :connection<Confirmed> })
}
}
}
}
}
}
sub sanitise( %edit --> Str ) {
my $response = 'OK';
for %edit.keys {
when 'editor' {
%edit<editor> .= subst(/\W/, '_', :g)
.substr( ^21 )
}
when 'patch' {
if %edit<patch>.chars > $patch-limit {
%edit<cancel> = True;
$response = 'TooManyChanges'
}
}
when 'comment' { %edit<comment> .= substr(^$comment-limit) }
when <sha repo content>.any {}
default { %edit{ $_ }:delete }
# remove unwanted fields
}
$response
}
sub raise-pr( %an-edit ) { ... }
The Server code handles all the interaction between the browser and the 'Hallowed Circle'.
Now we execute the remaining steps ( 2 - 5 ) of the Github sequence.
sub raise-pr( %edit ) {
#| Define the secret token. When the container is run,
#| the token can be passed as an environment parameter.
#| Later, we can refactor to obtain it to identify the
#| user making the edits.
my $id-token = %*ENV<ID_TOKEN>;
my $base = 'https://api.github.com/repos';
# create a descriptive branch name
my $branch = strftime(DateTime.now, '%v')
~ "_{ %edit<editor> }";
# get the repo information
my $resp = await Cro::HTTP::Client.get(
"/{%edit<repo>}/git/ref/heads/main"
);
my %data = await $resp.body;
my $commit = %data<object><sha>;
# create a new branch
$resp = await Cro::HTTP::Client.post(
"$base/{%edit<repo>}/git/refs",
content-type => 'application/vnd.github+json',
auth => { bearer => $id-token },
headers => [ X-GitHub-Api-Version => '2022-11-28' ],
body => %(:sha($commit), :ref("refs/heads/$branch"))
);
%data = await $resp.body;
# update file into new branch
$resp = await Cro::HTTP::Client.put(
"$base/{%edit<repo>}/contents/{%edit<path>}",
content-type => 'application/vnd.github+json',
auth => { bearer => $id-token },
headers => [ X-GitHub-Api-Version => '2022-11-28' ],
body => %(
:sha(%edit<sha>),
:$branch,
:content(%edit<content>),
:message(%edit<comment>),
)
);
%data = await $resp.body;
# should check to make sure OK
# Raise a PR for the new branch
$resp = await Cro::HTTP::Client.post(
"$base/{%edit<repo>}/pulls",
content-type => 'application/vnd.github+json',
auth => { bearer => $id-token },
headers => [ X-GitHub-Api-Version => '2022-11-28' ],
body => %(
:title("Web edit of {%edit<path>}"),
:head($branch),
:base<main>,
:body("Edit suggested by ｢{%edit<editor>}｣ because ｢{%edit<comment>｣}"),
)
);
}
The Github documentation for the PR was particularly confusing because head, base, and body did not seem to be intuitive taken on their own.
I have documented my journey into and out of the woods using as little jargon as possible. My purpose in doing this was to describe it so I could understand it myself, and also in the hope that the gods of the internet (aka search engines) will guide someone else who may have a similar set of issues.
Naturally, the container is part of a bigger system, and it needs to be tested. But that raises more questions, so I will only be brief.
It is not so easy to test a websocket. My approach was:
localhost.In real world we have a lot of pure structured or structured in hard to parse formats configuration files:
/etc/default/nginx, /etc/default/grub, logrotate.conf, sysctl.conf - to name a few.
In configuration management it's not only important to maintain specific configurations but also validate those changes to make it sure we don't break things.
Sparrow - is an automation tool written on Raku that allow to make such a validation by leveraging Task::Check DSL rules which underneath are just Raku regular expressions.
In the rest of this post we take a look at real life example.
Consider linux GRUB boot loader file /etc/default/grub with following configuration:
GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=2
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="rootfstype=ext4 quiet splash acpi_osi="
GRUB_CMDLINE_LINUX=""
GRUB_ENABLE_CRYPTODISK=true
# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console
# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true
# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"
# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
GRUB_DISABLE_OS_PROBER="true"
Though configuration format is quite simple, based on VAR=value semantics, it still allows some freedom in specification.
For example to enable variables we could use one of many forms - yes/"yes", true/"true" and just 1. Another challenge is to check that some variable are disabled - they might be just commented or “switched off” via negation form, e.g. VAR=0,false,""
As rigorous operations we want to validate a couple of things:
Boot on crypted disks is supported by setting GRUB_ENABLE_CRYPTODISK to one of the values: 1,yes,true and optional quoted form - "yes","true"
OS prober capability is disabled or in other words GRUB_DISABLE_OS_PROBER is not set by using any of forms - 1,yes,true,"yes","true"
Before go parsing let's create simple Raku program (Sparrow task), that dumps out configuration removing empty and commented lines, the whole output will be placed inside >>>, <<< markers for visibility:
task.raku
#!raku
my $path = config()<path>;
say "validate config {$path} ...";
say ">>>";
my @out;
for $path.IO.lines -> $i {
$i.chomp;
next unless $i ~~ /\S+/;
@out.push($i) unless $i ~~ /^ \s* '#'/;
};
say sort(@out).join("\n");
say "<<<";
Now let's write verification scenario by creating rules in Sparrow::Task::Check DSL format:
task.check
note: validate config
between: {'>>>'} {'<<<'}
note: allow to boot on crypted disks
regexp: ^^ \s* "GRUB_ENABLE_CRYPTODISK=" (<[y 1 true]> | '"true"' | '"yes"' )
note: never allow to enable os prober
!regexp: ^^ \s* "GRUB_DISABLE_OS_PROBER=" (<[y 1 true]> | '"true"' | '"yes"' )
end:
The DSL code has there major parts:
between: {'>>>'} {'<<<'}Setting a context for a search in between >>> ... <<< delimiter, this is an example of so called search context modifiers, in this case - Range modifier.
In other words this is precaution that we only search within specific boundaries of text output
regexp: ^^ \s* "GRUB_ENABLE_CRYPTODISK=" (<[y 1 true]> | '"true"' | '"yes"' )
Verifying that we have GRUB_ENABLE_CRYPTODISK set to one of values (y,1,true,"true","yes").
regexp: marker means Raku regular expression is used for validation.
note: never allow to enable os prober
!regexp: ^^ \s* "GRUB_DISABLE_OS_PROBER=" (<[y 1 true]> | '"true"' | '"yes"' )Verifying we DON'T(*) have GRUB_ENABLE_CRYPTODISK set to one of values (y,1,true,"true","yes"), in other words that GRUB_ENABLE_CRYPTODISK is disabled.
(*) ! sign before regexp: marker means negation logic.
Note: expressions are human readable comments that are showed up in a report to help us correlate every check to business logic
Now let's run our script and get a result of test. As this is Sparrow scenario, we use s6 - Sparrow CLI runner:
#!bash
s6 --task-run .@path=/etc/default/grub
14:09:20 :: [sparrowtask] - run sparrow task .@path=/etc/default/grub
14:09:20 :: [sparrowtask] - run thing .
[task run: task.raku - .]
[task stdout]
14:09:21 :: validate config examples/grub ...
14:09:21 :: >>>
14:09:21 :: GRUB_CMDLINE_LINUX=""
14:09:21 :: GRUB_CMDLINE_LINUX_DEFAULT="rootfstype=ext4 quiet splash acpi_osi="
14:09:21 :: GRUB_DEFAULT=0
14:09:21 :: GRUB_DISABLE_OS_PROBER="true"
14:09:21 :: GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
14:09:21 :: GRUB_ENABLE_CRYPTODISK=true
14:09:21 :: GRUB_HIDDEN_TIMEOUT_QUIET=true
14:09:21 :: GRUB_TIMEOUT=2
14:09:21 :: <<<
[task check]
# validate config
# allow to boot on crypted disks
stdout match (r) <^^ \s* "GRUB_ENABLE_CRYPTODISK=" (<[y 1 true]> | '"true"' | '"yes"' )> True
# never allow to enable os prober
stdout match (r) <!^^ \s* "GRUB_DISABLE_OS_PROBER=" (<[y 1 true]> | '"true"' | '"yes"' )> False
=================
TASK CHECK FAIL
Expectedly we have following results:
Pretty impressive results considering such a little amount of code has been written!
Sparrow Task::Checks is comprehensive and super flexible tool allowing one to write validation scenarios for arbitrary data formats and complexity, check out documentation pages to know more.
That is it. Thanks for reading
Recently, I've released 2 new Sparrow plugins to help build Golang code:
Simple golang code builder
Golang code formatter
They are simple yet useful, providing two essentials tools to build Golang code in terminal:
Build code
Format code
The rest of the post shows how to use the plugins
To install plugins we need to use Tomtit task runner with profiles - preset of Raku scenarios that grouped by topics:
#!bash
cd /to/you/source/code
tom --init # initialize Tomtit for the first time
tom --profile go # install go profile scenarios
Now once we've installed go profile, we have an access to go-build and go-format scenarios which are just a Raku wrappers to run mentioned plugins. Code is simple enough and editable to adjust your project specific needs
Go-build scenario:
#!bash
tom --cat go-build
#!raku
my $path = [
"cmd/app/main.go",
"cmd/cli/cli.go"
];
task-run "build", "go-build", %(
:$path,
);
Go-format scenario:
#!bash
tom --cat go-format
#!bash
task-run "go format", "go-format";
In case of go-build scenario - adjust path array with files to build for your project specific:
#!bash
export EDITOR=nano
tom --edit go-build
#!raku
my $path = [
"cmd/app/app.go",
"cmd/app2/app2.go",
];
# the rest of the code stays the same
To run plugins - use tom cli runner:
#!bash
tom go-build
11:56:26 :: go build cmd/main.go
11:56:27 :: cmd/main.go
#!bash
tom go-format
11:57:18 :: cmd/main.go
That is it.
Follow plugins documentation to get familiar with plugins parameters.
Thanks for reading
I published my new book: A Language a Day, which is a collection of brief overviews to 21 programming languages.
This book provides a concise overview of 21 different programming languages. Each language is introduced using the same approach: solving several programming problems to showcase its features and capabilities. Languages covered in the book: C++, Clojure, Crystal, D, Dart, Elixir, Factor, Go, Hack, Hy, Io, Julia, Kotlin, Lua, Mercury, Nim, OCaml, Raku, Rust, Scala, and TypeScript.
Each chapter covers the essentials of a different programming language. To make the content more consistent and comparable, I use the same structure for each language, focusing on the following mini projects:
Each language description follows—where applicable—this pattern:
You can find all the code examples in this book on GitHub: github.com/ash/a-language-a-day.
You can buy it on Amazon or LeanPub as an electronic or Kindle edition, or as a paper hardcover or paperback version. More information with the links to the shops.
Last Saturday, October 26, 2024, another edition of the London Perl & Raku Workshop took place. It was a reopening after a few years break. London Perl Workshop has a long history, started in 2004, so this year it was formally 20 years old.
For me, it was also a Perl-and-or-Raku event after a break. The last in-person event was PerlCon 2019, which was an annual European Perl conference that was held in Riga that year. Since then, after the Covid’s strike, I have organised a couple of (well, three of them) Raku Conferences, but that was online in Zoom. Online due to lockdown first, and then due to people slowly reinventing the joy of travel.
In-between, there were at least two in-person gatherings, another edition of the German Perl and Raku Workshop in April and the American Perl and Raku Conference in Las Vegas in June. I did not attend any of them, but looked at the photos and videos, so I was envy enough because I missed it.
You should not forget that right now, right today, at this very moment, an on-going war against my beloved Ukraine takes place. During the first months, it was not possible to think about anything else, not to mention any entertainment trips or conferences. But time keeps going… and you see less and less blue-and-yellow flags on the streets. Even at Waterstones, the books about that war are located at the very bottom shelves due to the unfortunate nature of alphabetical sorting.
When this year’s London Perl Workshop was announced, I decided to go. Then, I decided not to go. In the end, one week before the event, I thought that I’d better go. Because of my uncertainty, I did not manage to submit a talk on time, so even a small 5-minute lightning talk submission (which I did on the day of the event, ha-ha), did not have a chance to went through. The venue, apparently, was rented until 6pm sharp, and there was no room for anything else.
So, here are the slides of my talks. I’ve added some comments so that you can enjoy it by following my message.
My journey was quite short, I only had a couple of days in London, but the weather made its present and the plane from Amsterdam landed in Southampton instead. So, I took a chance to see the old walls there and to learn that they offer beans on breakfast at IKEA’s canteen.
I really enjoy London, it’s so huge that you fill extremely comfortable in it, how weird that may sound. Despite only a limited couple of days in London, I tried to use the time to wander around just as a tourist.
I also escaped from the middle of the conference day to see the urban life around venue’s location.
On the day before the conference, I met with an ex-colleague, with whom I worked at two different places, both Perl-related. For a few years, he’s been living in London, and I noticed that he looked so much happier since I last saw him. I hope that is because he lives in this great city now, not because he’s programming in Go, not Perl anymore.
Of course, I was happy to see people I know from the Perl and/or Raku communities. I did not see most of them at least since 2019, so some of them required a few seconds to recognise me, LOL.
The workshop was really well attended. Even taking into account the significant drop in the numbers since the beginning of 2000s, it was more than 100 people, and there were two full tracks of the talks.
Let me not list the talks I attended, as the full schedule is online, and the recordings should appear soon.
There was also a third track, partially filled with regular talks, but partially with a broader-scoped events such as Science Perl Talks. I’ve no idea what they talked about there, but it’s somehow connected with the recently-published first issue of the Science Perl Journal and a traditional drama around its establisher.
But the appearance of the new publication is a good thing in the end. I was also surprised, that in 2024, a few new publications about Perl appeared or are planned to be published. I suspect that some of them issued during the previous years are AI-generated texts, but the rest are quite normal and useful books.
There are no decision yet whether there will be another London Perl Workshop in 2025. On the bright side, there should be a soon announcement on the date and location of the German Perl Workshop 2025. Also, there are plans to have a Dutch Perl Workshop 2025 in Utrecht.
I am thinking of if it’s time to revitalise the annual European Perl and Raku Conference.
NOTE: There is an issue when opening existing Comma projects that were created in earlier versions. Please use New project from Existing Sources... rather than Open and make sure to select Yes when it prompts you about overwriting an existing .idea file in the project directory.
This release represents a major shift for the Comma project in many ways.
From the bottom of my heart, I want to express the deepest gratitude and thanks to Jonathan Worthington (jnthn++),
Edument, and all past and future contributors to the Comma project. There's been
so much effort put into this codebase and it was an honor to be able to work on it.
The most major change is the shift to the IntelliJ Platform Gradle Plugin 2.0. This
allows Comma to be built (as a plugin) without cloning the
intellij-community repo and downloading it's entire dependency tree!
This does seem to preclude building Comma as a standalone IDE, at least for the time being. That appears to be a different beast entirely and we will have to investigate that as the time and tuits allow.
Other major changes included updating the code to correct for broken and (some) deprecated
API changes, as well as the significant cosmetic adjustment of migrating Perl6 to Raku.
The latter should be almost entirely finished, but there might be some stragglers that I've
missed.
Building should be as simple as opening this repository in IntelliJ IDEA (using version 2024.2 or greater), and selecting
build > build from the Gradle build target options. Or, for more immediate gratification, you can select intellij platform > runIde.
Update: If you don't feel like building it yourself, you can now simply download the plugin zip from GitHub. From inside IntelliJ IDEA, open the Settings > Plugins, find the gear icon, and select Install Plugin from Disk....
Next steps:
<Your wishlists go here!>Happy hacking! :D
The question has been raised, how to get named arguments into sub EXPORT via a use-statement. The ever helpful raiph provided an answer, which in turn left me with the question, why he didn’t just use a Capture to move the data around. Well, because that doesn’t work. The compiler actually evaluates the expression \(:1a, :2b) into (1, 2) before passing it on to EXPORT.
If it’s hard, do it functional!
# foo.raku
use v6.d;
constant &transporter = sub { \(:1a, :2b); }
use foo &transporter;
# lib/foo.rakumod
use v6.d;
proto sub EXPORT(|) { * }
multi sub EXPORT(&transporter) {
&EXPORT(|transporter);
}
multi sub EXPORT(:$a, :$b) {
dd $a, $b;
Map.new
}The idea is to hand a function to use to be called by EXPORT, and then redispatch the value that is produced by that function, to take advantage of Raku´s excellent signature binding. The proto and refering to sub EXPORT explicitly is needed because there is also a predefined (and in this case hidden) package called EXPORT.
I’m passing on named arguments to EXPORT, but all kinds of stuff could be returned by &transporter. So long as everything is known pretty early on at compile-time. The use-statement is truly an early bird.
As the title states, I made Raku bigger because lol context (that’s how the Synopsis is calling **@) makes supporting feed operators fairly easy. I wonder if Larry added this syntax to Signature with that goal in mind. With PR#5532 the following becomes possible.
<abc bbc cbc> ==> trans('a' => 'x', 'b' => 'i') ==> say();
# OUTPUT: (xic iic cic)Armed with this I can make a script of mine a little simpler.
use MONKEY-TYPING;
augment class IO::Path {
method trans(|c) {
my $from = self.Str;
my $to = self.Str.trans(|c);
self.rename($to) unless $from eq $to
}
}
sub rename-whitespace(IO::Path $dir where *.d){
dir($dir).grep({ .d || .f && .rw })
==> trans("\c[space]" => "\c[no-break space]", "\c[apostrophe]" => "\c[prime]")
==> sub (*@a) { print '.' for @a}();
dir($dir).grep({ .d && .rw })».&?ROUTINE;
}
rename-whitespace('.'.IO);
put '';I don’t like spaces in filenames, as they are often found with audio or video files. Having auto-complete friendly names makes using a CLI less bumpy. By teaching IO::Path to rename files by providing rules, as they are understood by Str.trans, I can use a feed operator to get the job done. (I wouldn’t be surprised to learn, that anonymous subs DWIM here to be emergent behaviour in Raku.)
Having another PR that adds .trans to IO::Path is tempting but requires more thought.
A follow up to the Welsh dragon.
Firing up another localisation
Steps to Ryuu
Comments on the Raku program
More generally about localisation of coding
If you want to make Ryuu better?
In my previous blog about Y Ddraig, I created a localisation of the Raku Language in Welsh. During a recent conversation, someone mentioned there may be interest in a Japanese localisation, so I thought I would try the same techniques.
I do not speak or read or have ever studied Japanese. The localisation given below will be about as clunky and awkward as any can be. I imagine there may be some hilarious stupidities as well.
So to be clear, this article is about a proof of concept rather than a real effort to create a production-ready program.
However, it took me 40 minutes from start to finish, including setting up the github repo.
Since I like dragons, I named the Japanese cousin to Raku 'Ryuu'. It's a whimsy, not to be treated with much seriousness.
Basically I created a github repo, copied my existing Welsh localisation and changed CY to JA, and draig to ryuu.
Within the automation/ directory I used the translation technique explained for Welsh to create the JA file from the template. The translated.txt file needed some manual cleaning, because the English word for has multiple Japanese equivalents. I chose one more or less at random. In addition, Google translate did some strange things to the order of words and numbers in a line.
After adapting the files in the bin/ directory, and installing the distribution with Raku's zef utility, I ran tr2ryuu on the Raku program simple.raku.
A comment about my Welsh blog was that the program in Y Ddraig was not all in Welsh. And here the program is not all in Japanese.
Remember that the user-facing part of a program will be in the language of the user, in this case it is English. However, the coder-facing part of the program will be in the language of the coder. Below, the coder interface is in Japanese (or rather my ham-fisted attempt at Japanese).
The following is the result (which I put in a file called simple.ryuu):
私の $choice;
私の $continue;
私の @bad = <damn stupid nutcase>;
リピート {
$choice = プロンプト "Type something, like a number, or a string: ";
言う "You typed in ｢" ~ ($choice ~~ 任意(@bad) ?? "*" × $choice.文字 !! $choice) ~ "｣";
与えられた $choice {
いつ "dragon" {
言う "which is 'draig' in Welsh"
}
いつ 任意(@bad) {
言う "wash your mouth with soap"
}
いつ IntStr {
言う "which evaluates to an integer ", $choice
}
いつ RatStr {
言う "which evaluates to a rational number ", $choice
}
デフォルト {
言う "which does not evaluate to a number "
}
}
$continue = プロンプト "Try again? If not type N: "
} まで $continue 当量 任意(<N n>)
What is amazing to me is that when I ran ryuu simple.ryuu, the program ran without error.
The simple.raku program is obviously trivial, but what I wanted to show are some interesting Raku features. Note how I created an array of words with @bad = <damn stupid nutcase>;, and then later I tested to see whether an input word was one of the array elements.
The Raku idiom いつ 任意(@bad) or in English when any( @bad ) compares the topic variable, in this case the input value, with each array element and creates a junction of Boolean results. The 'any' effectively or's the result to collapse the junction.
Junctions are not common in programming languages, so I thought if there would be problems, then it would be there. So I was surprised to find my Raku program works without error in another language.
All the major coding languages are in English. There are, however, coders from all over the world, and the majority of those from non-English speaking nations would have needed to learn English before (or at the same time as) they learnt coding.
We are thus creating a new technological elite: those who can understand English (or some subset of it), and those who cannot. The more coding becomes an essential part of life, the greater the ability divide between coders (who speak English) and non-coders will become.
The aim of localising a programming language is to provide an entry into coding in a form that is more accessible to every human being, whatever their natural language.
However, the aim of this approach is not to eliminate English at every level of complexity, but to provide a sufficiently rich language for most normal coding and educational needs.
In addition, by having a canonical language (Raku, which is based on English) into which all localised languages can be translated, what we get is a universal auxiliary language together with a universality of being able to code.
Having a single auxiliary language means that a non-English speaking person writing in a localised coding language can translate the program with the problem into Raku, have a developer on the other side of the globe find the problem, and suggest a solution in code, then for that solution to be translated back into the local language.
Naturally, a person who wants to learn more about coding, or who needs to delve deeper into the workings of a module, will need to learn English. Learning wider to learn deeper is a normal part of the educational experience.
Ryuu or however it should be called, absolutely is in need of Tender loving care. Please feel free to use the github issues or PR processes to suggest better translations.
At some stage, Ryuu will join the official Raku localisations.
Over on Reddit zeekar wasn’t too happy about Raku’s love of Seq. It’s immutability can be hindering indeed.
my @nums = [ [1..10], ];
@nums[0] .= grep: * % 2;
@nums[0].push(11); # We can't push to a Seq.I provided a solution I wasn’t happy with. It doesn’t DWIM and is anything but elegant. So while heavily digesting on my sofa (it is this time of the year), the problem kept rolling around in my head. At first I wanted to wrap Array.grep(), but that would be rather intrusive and likely break Rakudo itself. After quite a bit of thinking, I ended up with the question. How can I have indexable container (aka Array) that will turn each value on assignment into an (sub-)Array?
my Array() @foo = [ 1..10, ];
dd @foo;
# Array[Array(Any)] @foo = Array[Array(Any)].new($[1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
@foo[0] .= grep: * % 2;
@foo[1] = 42;
dd @foo;
# Array[Array(Any)] @foo = Array[Array(Any)].new($[1, 3, 5, 7, 9], $[42])The answer is obvious. By telling the compiler what I want! Coersion-types have become really hard to distinguish from magic.
I wish you all a Merry Christmas and the very best questions for 2024.
Note: This post is also available as a gist if you find that format more readable.
This research was conducted while preparing an upcoming Raku Advent Calendar post. The Raku code uses a basic supply pipeline to feed $volume objects through a validation stage that requires a CRC32 check before going to the output sink, which prints the processing time of the validation stage.
The "reaction graph" is designed to simulate a stream processing flow, where inputs arrive and depart via Candycane™ queues (that's the name of Santa's Workshop Software's queueing service, in case you weren't familiar).
The entire scenario is contrived in that CRC32 was chosen due to native implementation availability in both Raku and Zig, allowing comparison. It's not an endorsement of using CRC32 in address validation to deliver Santa's, or anyone's, packages.
Also, thanks to the very helpful folks at ziggit.dev for answering my newbie question in depth.
The source code:
At larger volumes, Raku struggles with the initialization speed of the $volume objects that are instantiated. I replaced the native Raku class with one written in Zig, using the is repr('CStruct') trait in Raku and the extern struct qualifier in Zig.
In Zig I use a combination of an arena allocator (for the string passed from Raku) and a memory pool (designed to quicklymake copies of a single type, exactly fitting our use case) to construct Package objects.
Additionally, for Raku+Zig the CRC32 hashing routine from Zig's stdlib is used via a tiny wrapper function.
A --bad-packages option is provided by both Raku scripts, which makes 10% of the objects have a mismatched address/CRC32 pair.
The library tested was compiled with -Doptimize=ReleaseFast.
Batches are repeated $batch times, which defaults to 5.
All results from an M2 MacBook Pro.
This test and its is only intended to reflect the case where an object is constructed in Zig based on input from Raku. It is not intended to be a test of Zig's native speed in the creation of structs.
There is a call to sleep that gives time -- 0.001 seconds -- to get the react block up and running before emitting the first True on the $ticker-supplier. This affects overall runtime but not the batch or initialization metrics.
The speed of Raku+Zig was so fast that the tool used to measure these details (cmdbench) could not find results in ps for the execution because it had already finished. These are marked as Unmeasured.
In the next iteration of this research, there sould be two additional entries in the data tables below for:
| Volume | Edition | Runtime | Batch Time | Initialization | Max bytes |
|---|---|---|---|---|---|
| 10,000 | Raku | 1.072s | 1: 0.146596686s 2: 0.138983732s 3: 0.142380065s 4: 0.136050775s 5: 0.134760525s |
0.008991746s | 180240384 |
| 10,000 | Raku+Zig | 0.44s | 1: 0.010978411s 2: 0.006575705s 3: 0.004145623s 4: 0.004280415s 5: 0.00468929s |
0.020358033s | Unmeasured |
| 10,000 | Raku ( bad-packages) |
1.112s | 1: 0.157788932s 2: 0.149544686s 3: 0.156293433s 4: 0.151365477s 5: 0.147947436s |
0.008059955s | 196263936 |
| 10,000 | Raku+Zig ( bad-packages) |
0.463s | 1: 0.031300276s 2: 0.01006562s 3: 0.010693328s 4: 0.011056994s 5: 0.010770828s |
0.010954495s | Unmeasured |
The Raku+Zig solution wins in performance, but loses the initialization race. Raku is doing a decent showing in comparison to how far it has come performance-wise.
| Volume | Edition | Overall | Batch Time | Initialization | Max bytes |
|---|---|---|---|---|---|
| 100,000 | Raku | 7.163s | 1: 1.360029456s 2: 1.32534014s 3: 1.353072834s 4: 1.346668338s 5: 1.351110502s |
0.062402473s | 210173952 |
| 100,000 | Raku+Zig | 0.75s | 1: 0.079802007s 2: 0.073638176s 3: 0.053291894s 4: 0.05087652s 5: 0.050394687s |
0.05855585s | 241205248 |
| 100,000 | Raku ( bad-packages) |
7.89s | 1: 1.496982355s 2: 1.484494027s 3: 1.497365023s 4: 1.490810525s 5: 1.492416774s |
0.060026016s | 209403904 |
| 100,000 | Raku+Zig ( bad-packages) |
1.076s | 1: 0.16960934s 2: 0.111172493s 3: 0.110844786s 4: 0.113021202s 5: 0.111713535s |
0.051436311s | 242450432 |
We see Raku+Zig take first place in everything but memory consumption, which we can assume is a function of using the NativeCall bridge, not to mention my new-ness as a Zig programmer.
| Volume | Edition | Overall | Batch Time | Initialization | Max bytes |
|---|---|---|---|---|---|
| 1,000,000 | Raku | 68.081s | 1: 13.475302627s 2: 13.161153845s 3: 13.293998956s 4: 13.364662217s 5: 13.474755295s |
0.95481884s | 417103872 |
| 1,000,000 | Raku+Zig | 3.758s | 1: 0.788083286s 2: 0.509883905s 3: 0.492898873s 4: 0.500868284s 5: 0.498677495s |
0.575087671s | 514064384 |
| 1,000,000 | Raku+Zig ( bad-packages) |
75.796s | 1: 14.940173822s 2: 14.632683637s 3: 14.866796226s 4: 15.272903792s 5: 15.027481448s |
0.704549212s | 396656640 |
| 1,000,000 | Raku+Zig ( bad-packages) |
6.553s | 1: 1.362189763s 2: 1.061496504s 3: 1.069134685s 4: 1.062746049s 5: 1.061096044s |
0.528011288s | 462766080 |
Raku's native CRC32 performance is clearly lagging here. Raku+Zig keeps its domination except in the realm of memory usage. It would be hard to justify using the Raku native version strictly on its reduced memory usage, considering the performance advantage on display here
A "slow first batch" problem begins to affect Raku+Zig. Running with bad-packages enabled slows down the Raku+Zig crc32 loop, hinting that there might be some optimizations on either the Raku or the Zig/clang side of things that can't kick in when the looped data is heterogenous.
Dynamic runtime optimization sounds more like a Rakudo thing than a Zig thing, though.
| Volume | Edition | Runtime | Batch Time | Initialization | Max bytes |
|---|---|---|---|---|---|
| 10,000,000 | Raku | 704.852s | 1: 136.588638184s 2: 136.851019628s 3: 138.44696743s 4: 139.777040922s 5: 139.490784317s |
13.299274221s | 2055012352 |
| 10,000,000 | Raku+Zig | 38.505s | 1: 8.843459877s 2: 4.84300835s 3: 4.991842433s 4: 5.077245603s 5: 4.939533707s |
9.375436134s | 2881126400 |
| 10,000,000 | Raku ( bad-packages) |
792.1s | 1: 162.333803401s 2: 174.815386318s 3: 168.299796081s 4: 162.643428135s 5: 163.205406678s |
10.252639311s | 2124267520 |
| 10,000,000 | Raku+Zig ( bad-packages) |
65.174 | 1: 14.41616445s 2: 11.078961309s 3: 10.662389991s 4: 11.20240076s 5: 10.614430063s |
6.778600235s | 2861596672 |
Pure Raku really struggles with a volume of this order of magnitude. But if you add in just a little bit of Zig, you can reasonably supercharge Raku's capabilities.
The "slow first batch" for Raku+Zig has been appearing in more understated forms in other tests. Here the first batch is over double the runtime of the second batch. What is causing this?
This doesn't seem to work. At least, I'm not patient enough. The process seems to stall, growing and shrinking memory but never finishing.
This is a preliminary report in blog post form based on a contrived code sample written for another, entirely different blog post. More data and deeper analysis will have to come later.
Zig's C ABI compatibility is clearly no put on. It works seamlessly with Raku's NativeCall. Granted, we haven't really pushed the boundaries of what the C ABI can look like but one of the core takeaways is actually that with Zig we can design that interface. In other words, we are in charge of how ugly, or not, it gets. Considering how dead simple the extern struct <-> is repr('CStruct') support is, I don't think the function signatures need to get nearly as gnarly as they get in C.
Sussing the truth of that supposition out will take some time and effort in learning Zig. I'm looking forward to it. My first stop will probably be a JSON library that uses Zig. I'm also going to be looking into using Zig as the compiler for Rakudo, as it might simplify our releases significantly.
According to Larry, laziness is a programmers virtue. The best way to be lazy is having somebody else do it. By my request, SmokeMachine kindly did so. This is not fair. We both should have been lazy and offload the burden to the CORE-team.
Please consider the following code.
my @many-things = (1..10).List;
sub doing-one-thing-at-a-time($foo) { ... }
say doing-one-thing-at-a-time(@many-things.all);Rakudo goes out of it’s way to create the illusion that sub doing-one-thing-at-a-time can deal with a Junction. It can’t, the dispatcher does all the work of running code in parallel. There are tricks we can play to untangle a Junction, but there is no guarantee that all values are produced. Junctions are allowed to short-circuit.
This was bouncing around in my head for quite some time, until it collided with my thoughts about Range. We may be handling HyperSeq and RaceSeq wrong.
my @many-things = (1..10).List;
sub doing-one-thing-at-a-time($foo) { ... }
say doing-one-thing-at-a-time(@many-tings.hyper(:degree<10>));As with Junctions doing dispatch-magic to make hyper/race just work, moving the handling to the dispatcher would move the decision from the callee to the caller and, as such, from the author of a module to the user. We can do that by hand already with .hyper.grep(*.foo) or other forms of boilerplate. In Raku-land we should be able to do better and provide a generalisation of transforming calls with the help of the dispatcher.
I now know what to ask Santa for this year.
My version of JSON::Class is now released. The previous post explains why does this worth a note.
Lately, some unhappiness has popped up about Range and it’s incomplete numericaliness. Having just one blogpost about it is clearly not enough, given how big Ranges can be.
say (-∞..∞).elems;
# Cannot .elems a lazy list
in block <unit> at tmp/2021-03-08.raku line 2629I don’t quite agree with Rakudo here. There are clearly ∞ elements in that lazy list. This could very well be special-cased.
The argument has been made, that many operators in Raku tell you what type the returned value will have. Is that so? (This question is always silly or unnecessary.)
say (1 + 2&3).WHAT;
# (Junction)Granted, Junction is quite special. But so are Ranges. Yet, Raku covers the former everywhere but the latter feels uncompleted. Please consider the following code.
multi sub infix:<±>(Numeric \n, Numeric \variance --> Range) {
(n - variance) .. (n + variance)
}
say 2.6 > 2 ± 0.5;
# True
my @heavy-or-light = 25.6, 50.3, 75.4, 88.8;
@heavy-or-light.map({ $_ ≤ 75 ± 0.5 ?? „$_ is light“ !! „$_ is heavy“ }).say;
# (25.6 is heavy 50.3 is heavy 75.4 is heavy 88.8 is heavy)To me that looks like it should DWIM. It doesn’t, because &infix:«≤» defaults to coercing to Real and then comparing numerically.
This could easily be fixed by adding a few more multis and I don’t think it would break any production code. We already provide quite a few good tools for scientists. And those scientists do love their error bars — which are ranges. I would love for them to have another reason to use Raku over … that other language.
This will be a short one. I have recently released a family of WWW::GCloud modules for accessing Google Cloud services. Their REST API is, apparently, JSON-based. So, I made use of the existing JSON::Class. Unfortunately, it was missing some features critically needed for my work project. I implemented a couple of workarounds, but still felt like it’s not the way it has to be. Something akin to LibXML::Class would be great to have…
There was a big “but” in this. We already have XML::Class, LibXML::Class, and the current JSON::Class. All are responsible for doing basically the same thing: de-/serializing classes. If I wanted another JSON serializer then I had to take into account that JSON::Class is already taken. There are three ways to deal with it:
JSON::Class and re-implement it as a backward-incompatible version.The first two options didn’t appeal to me. The third one is now about to happen.
I expect it to be a stress-test for Raku ecosystem as, up to my knowledge, it’s going to be the first case where two different modules share the same name but not publishers.
As a little reminder:
JSON::Class:auth<zef:jonathanstowe> in their dependencies and, perhaps, in their use statement.JSON::Class:auth<zef:vrurg>.There is still some time before I publish it because the documentation is not ready yet.
Let’s 🤞🏻.
In this post I will show you the solution to the Task 2 of this week’s Weekly Challenge 233. This time I was using the help of ChatGPT, which offered a very nice solution.
You are given an array of integers.
Write a script to sort the given array in increasing order based on the frequency of the values. If multiple values have the same frequency then sort them in decreasing order.
Example 1
Input: @ints = (1,1,2,2,2,3)
Ouput: (3,1,1,2,2,2)
'3' has a frequency of 1
'1' has a frequency of 2
'2' has a frequency of 3
Example 2
Input: @ints = (2,3,1,3,2)
Ouput: (1,3,3,2,2)
'2' and '3' both have a frequency of 2, so they are sorted in decreasing order.
Example 3
Input: @ints = (-1,1,-6,4,5,-6,1,4,1)
Ouput: (5,-1,4,4,-6,-6,1,1,1)I asked ChatGPT the following: ‘Create a program. Here is the description of the task.’ To make sure the result is in the Raku programming language, I also added the following instruction in my local settings so that it knows which programming language I prefer by default.
The solution generated by ChatGPT was quite good with a small mistake.
ChatGPT: Sure, here’s a Raku program that implements the described sorting algorithm for the given task:
sub frequency-sort(@arr) {
my %freq;
%freq{$_}++ for @arr;
my @sorted = @arr.sort: {
%freq{$^b} <=> %freq{$^a} ||
$^b <=> $^a
};
return @sorted;
}
my @ints = (1, 1, 2, 2, 2, 3);
my @sorted = frequency-sort(@ints);
say "Input: @ints";
say "Output: @sorted";
First, Raku would not interpolate arrays, so say "Input: @ints" prints the string literary: Input: @ints. Second, in the sorting rule, sorting by frequency must be ascending rather than descending, so we need to switch the placeholder variables $^b and $^a.
In the rest, ChatGPT did a good work. It computed the frequency using a loop, and I especially liked how it addressed the requirement to sort by two dimensions.
Here is the full program with the above-mentioned corrections:
sub frequency-sort(@arr) {
my %freq;
%freq{$_}++ for @arr;
my @sorted = @arr.sort: {
%freq{$^a} <=> %freq{$^b} ||
$^b <=> $^a
};
return @sorted;
}
my @tests =
(1, 1, 2, 2, 2, 3),
(2, 3, 1, 3, 2),
(-1, 1, -6, 4, 5, -6, 1, 4, 1);
say frequency-sort($_) for @tests;
This program prints the following output for the given test cases:
$ raku ch-2.raku [3 1 1 2 2 2] [1 3 3 2 2] [5 -1 4 4 -6 -6 1 1 1]
In this post, I will demonstrate my solution to another Task of The Weekly Challenge, week 233. Here’s how it reads:
You are given an array of words made up of alphabets only.
Write a script to find the number of pairs of similar words. Two words are similar if they consist of the same characters.
Example 1
Input: @words = ("aba", "aabb", "abcd", "bac", "aabc")
Output: 2
Pair 1: similar words ("aba", "aabb")
Pair 2: similar words ("bac", "aabc")
Example 2
Input: @words = ("aabb", "ab", "ba")
Output: 3
Pair 1: similar words ("aabb", "ab")
Pair 2: similar words ("aabb", "ba")
Pair 3: similar words ("ab", "ba")
Example 3
Input: @words = ("nba", "cba", "dba")
Output: 0There’s a slight moment that may be needs extra comments. In the second example all three words constructed of the same two letters, a and b. So, all of the three words match the definition of a ‘similar’ word. But as the task needs to find pairs, we need to construct all the possible pairs out of those three words.
In my solution, I chose to use a handy classify method. For an array, it creates a hash, where the keys are the common classifying symbol, and the values are the lists of the input elements that match this classification property.
Here is the whole first program together with all the test cases provided in the description. The program maps every word to a corresponding string that consists of the sorted unique letters in the word.
my @tests = ["aba", "aabb", "abcd", "bac", "aabc"],
["aabb", "ab", "ba"],
["nba", "cba", "dba"];
for @tests -> @words {
say @words.classify(*.comb.unique.sort.join).grep(*.value.elems > 1);
}
For example, the word aba will be associated with the key ab. The program prints the following output:
$ raku ch-1.raku (ab => [aba aabb] abc => [bac aabc]) (ab => [aabb ab ba]) ()
The format of the output differs from the examples, but it can be enhanced if needed. My goal was to create a compact solution 
But I would assume that you’d be interested in looking at what classify produces. I am also curious. For the same @tests, it returns the following three hashes:
{ab => [aba aabb], abc => [bac aabc], abcd => [abcd]}
{ab => [aabb ab ba]}
{abc => [cba], abd => [dba], abn => [nba]}
As you see, each string was put into one of the classification bins.
The second part of the task is to find pairs. After the grep, we already filtered out everything that has less than two elements, so if data passed through this filter, there will be at least one pair. For bigger arrays, we can use another Raku’s built-in mechanism: the combinations method.
The updated mail loop of the program looks like this now.
for @tests -> @words {
say "Test case: ", @words;
my %classification = @words.classify(*.comb.unique.sort.join).grep(*.value.elems > 1);
my $pairs = 0;
for %classification.kv -> $k, $v {
my @pairs = $v.combinations(2);
$pairs += @pairs.elems;
say "$k: ", @pairs;
}
say "Answer: $pairs pair{$pairs == 1 ?? '' !! 's'}.\n";
}
The ‘redundant’ code here is added just to have a more detailed output so that we can see which pairs were actually found. Let us look at the output for the initial test cases:
$ raku ch-1.raku Test case: [aba aabb abcd bac aabc] ab: [(aba aabb)] abc: [(bac aabc)] Answer: 2 pairs. Test case: [aabb ab ba] ab: [(aabb ab) (aabb ba) (ab ba)] Answer: 3 pairs. Test case: [nba cba dba] Answer: 0 pairs.
On this page, I’ll briefly cover the solutions to the tasks for this week’s Weekly Challenge #231.
You are given an array of distinct integers.
Write a script to find all elements that is neither minimum nor maximum. Return -1 if you can’t.
Example 1
Input: @ints = (3, 2, 1, 4)
Output: (3, 2)
The minimum is 1 and maximum is 4 in the given array. So (3, 2) is neither min nor max.
Example 2
Input: @ints = (3, 1)
Output: -1
Example 3
Input: @ints = (2, 1, 3)
Output: (2)
The minimum is 1 and maximum is 3 in the given array. So 2 is neither min nor max.Here is my original solution in the Raku programming language.
sub solve(@data) {
@data.grep: * != (@data.min, @data.max).any
}
As the tasks requires that we print -1 when there are no elements in the output, let us add an update to satisfy this requirement:
sub solve(@data) {
(@data.grep: * != (@data.min, @data.max).any) || -1
}
The * in this code will actually replace the $_ variable. Would you prefer it, you may use $_, but you’ll need parentheses in this case. So, instead of @data.grep: * != ..., you need @data.grep({$_ != ...}), which may be a less clear code for some people.
Finally, let us use some math notation and replace calling the .any method with a ‘contains’ operator:
sub solve(@data) {
(@data.grep: * ∉ (@data.min, @data.max)) || -1
}
Well, actually, ‘does not contain’. And this is my final solution.
Note that you may want to use the .minmax method instead of two calls to .min and .max, but .minmax returns a range, which is not that suitable for this task.
Adding some test cases and passing them to the solve function:
my @tests = (3, 2, 1, 4), (3, 1), (2, 1, 3); say solve($_) for @tests;
The program prints the expected output:
$ raku ch-1.raku (3 2) -1 (2)
You are given a list of passenger details in the form “9999999999A1122”, where 9 denotes the phone number, A the sex, 1 the age and 2 the seat number.
Write a script to return the count of all senior citizens (age >= 60).
Input: @list = ("7868190130M7522","5303914400F9211","9273338290F4010")
Ouput: 2
The age of the passengers in the given list are 75, 92 and 40.
So we have only 2 senior citizens.
Input: @list = ("1313579440F2036","2921522980M5644")
Ouput: 0Apparently, the solution requires extracting information from a string in a specific format. It is not quite clear from the description whether the strings always contains the same number of characters, and thus the age and seat number are always two-digit values. But let’s use this assumption.
As we do not need any other information from the ticket code, no need to properly parse it, so I preferred anchoring around the only letter in the string and consider the next two digits as the age. Of course, you may make it simpler and just extract the two digits counting from the end of the string.
sub is-sinior($ticket) {
~($ticket ~~ / <alpha> (\d\d) /)[0] >= 75
}
Unlike Perl 5, Raku ignores spaces in regexes by default, so I added some air to it. On the other hand, extracting matches may seem a bit more complicated.
For the first given example (see task’s description), the Match object contains the following information:
｢M75｣ alpha => ｢M｣ 0 => ｢75｣
So, I am taking the 0th element using [0] and stringily it with the ~ prefix operator.
In essence, the task has been solved. Let’s add the test cases and run them:
my @tests = ('7868190130M7522', '5303914400F9211', '9273338290F4010'),
('1313579440F2036', '2921522980M5644');
for @tests -> @tickets {
say [email protected]({is-sinior($_)});
}
The program prints:
$ raku ch-2.raku 2 0
* * *
I was always concerned about making things easier.
No, not this way. A technology must be easy to start with, but also be easy in accessing its advanced or fine-tunable features. Let’s have an example of the former.
This post is a quick hack, no proof-reading or error checking is done. Please, feel free to report any issue.
Part of my ongoing project is to deal with JSON data and deserialize it into Raku classes. This is certainly a task
for JSON::Class. So far, so good.
The keys of JSON structures tend to use lower camel case which is OK, but we like
kebabing in Raku. Why not, there is
JSON::Name. But using it:
There are roles. At the point I came to the final solution I was already doing something like1:
class SomeStructure does JSONRecord {...}
Then there is AttrX::Mooish, which is my lifevest on many occasions:
use AttrX::Mooish;
class Foo {
has $.foo is mooish(:alias<bar>);
}
my $obj = Foo.new: bar => "the answer";
say $obj.foo; # the answer
Apparently, this way it would still be a lot of manual interaction with aliasing, and that’s what I was already doing for a while until realized that there is a bettter way. But be back to this later…
And, eventually, there are traits and MOP.
That’s the easiest part. What I want is to makeThisName look like make-this-name. Ha, big deal!
unit module JSONRecord::Utils;
our sub kebabify-attr(Attribute:D $attr) {
if $attr.name ~~ /<.lower><.upper>/ {
my $alias = (S:g/<lower><upper>/$<lower>-$<upper>/).lc given $attr.name.substr(2);
...
}
}
I don’t export the sub because it’s for internal use mostly. Would somebody need it for other purposes it’s a rare case where a long name like JSONRecord::Utils::kebabify-attr($attr) must not be an issue.
The sub is not optimal, it’s what I came up with while expermineting with the approach. The number of method calls and regexes can be reduced.
I’ll get back later to the yada-yada-yada up there.
Now we need a bit of MOP magic. To handle all attributes of a class we need to iterate over them and apply the aliasing. The first what comes to mind is to use role body because it is invoked at the early class composition times:
unit role JSONRecord;
for ::?CLASS.^attributes(:local) -> $attr {
# take care of it...
}
Note the word “early” I used above. It actually means that when role’s body is executed there are likely more roles waiting for their turn to be composed into the class. So, there are likely more attributes to be added to the class.
But we can override Metamodel::ClassHOW compose_attributes method of our target ::?CLASS and rest assured no one would be missed:
unit role JSONRecordHOW;
use JSONRecord::Utils;
method compose_attributes(Mu \obj, |) {
for self.attributes(obj, :local) -> $attr {
# Skip if it already has `is mooish` trait applied – we don't want to mess up with user's intentions.
next if $attr ~~ AttrX::Mooish::Attribute;
JSONRecord::Utils::kebabify-attr($attr);
}
nextsame
}
Basically, that’s all we currently need to finalize the solution. We can still use role’s body to implement the key elements of it:
unit role JSONRecord;
use JSONRecordHOW;
unless ::?CLASS.HOW ~~ JSONRecordHOW {
::?CLASS.HOW does JSONRecordHOW;
}
Job done! Don’t worry, I haven’t forgot about the yada-yada-yada above!
But…
The original record role name itself is even longer than JSONRecord, and it consists of three parts. I’m lazy. There are a lot of JSON structures and I want less typing per each. A trait? is jrecord?
unit role JSONRecord;
multi sub trait_mod:<is>(Mu:U \type, Bool:D :$jrecord) is export {
unless type.HOW ~~ JSONRecordHOW {
type.HOW does JSONRecordHOW
type.^add_role(::?ROLE);
}
}
Now, instead of class SomeRecord does JSONRecord I can use class SomeRecord is jrecord. In the original case the win is even bigger.
There is absolutely nothing funny about it. Just a common way to keep a reader interested!
Seriously.
The reason for the yada in that snippet is to avoid a distraction from the primary purpose of the example. Here is what is going on there:
I want AttrX::Mooish to do the dirty work for me. Eventually, what is needed is to apply the is mooish trait as shown above. But the traits are just subs. Therefore all is needed now is to:
&trait_mod:<is>($attr, :mooish(:$alias));
Because this is what Raku does internally when encounters is mooish(:alias(...)). The final version of the kebabifying sub is:
our sub kebabify-attr(Attribute:D $attr) {
if $attr.name ~~ /<.lower><.upper>/ {
my $alias = (S:g/<lower><upper>/$<lower>-$<upper>/).lc given $attr.name.substr(2);
&trait_mod:<is>($attr, :mooish(:$alias));
}
}
Since the sub is used by the HOW above, we can say that the &trait_mod<is> would be called at compile time2.
Now, it used to be:
class SomeRecord does JSONRecord {
has $.aLongAttrName is mooish(:alias<a-long-attr-name>);
has $.shortname;
}
Where, as you can see, I had to transfer JSON key names to attribute names, decide where aliasing is needed, add it, and make sure no mistakes were made or attributes are missed.
With the above rather simple tweaks:
class SomeRecord is jrecord {
has $.aLongAttrName;
has $.shortname;
}
Job done.
Before I came down to this solution I’ve got 34 record classes implemented using the old approach. Some are little, some are quite big. But it most certainly could’ve taken much less time would I have the trait at my disposal back then…
I have managed to finish one more article in the Advanced Raku For Beginners series, this time about type and object composition in Raku.
It’s likely to take a long before I can write another.
Once, long ago, coincidentally a few people were asking the same question: how do I get a method object of a class?
Answers to the question would depend on particular circumstances of the code where this functionality is needed. One
would be about using MOP methods like .^lookup, the other is to use method name and indirect resolution on invocant:
self."$method-name"(...). Both are the most useful, in my view. But sometimes declaring a method as our can be
helpful too:
class Foo {
our method bar {}
}
say Foo::<&bar>.raku;
Just don’t forget that this way we always get the method of class Foo, even if a subclass overrides method bar.
The first Raku Core Summit, a gathering of folks who work on “core” Raku things, was held on the first weekend of June, and I was one of those invited to attend. It’s certainly the case that I’ve been a lot less active in Raku things over the last 18 months, and I hesitated for a moment over whether to go. However, even if I’m not so involved day to day in Raku things at the moment, I’m still keen to see the language and its ecosystem move forward, and – having implemented no small amount of the compiler and runtime since getting involved in 2007 – I figured I’d find something useful to do there!
The area I was especially keen to help with is RakuAST, something I started, and that I’m glad I managed to bring far enough that others could see the potential and were excited enough to pick it up and run with it.
One tricky aspect of implementing Raku is the whole notion of BEGIN time (of course, this is also one of the things that makes Raku powerful and thus is widely used). In short, BEGIN time is about running code during the compile time, and in Raku there’s no separate meta-language; anything you can do at runtime, you can (in principle) do at compile time too. The problem at hand was what to do about references from code running at compile time to lexically scoped symbols in the surrounding scope. Of note, that lexical scope is still being compiled, so doesn’t really exist yet so far as the runtime is concerned. The current compiler deals with this by building up an entire flattened table of everything that is visible, and installing it as a fake outer scope while running the BEGIN-time code. This is rather costly, and the hope in RakuAST was to avoid this kind of approach in general.
A better solution seemed to be at hand by spotting such references during compilation, resolving them, and fixating them – that is, they get compiled as if they were lookups into a constant table. (This copies the suggested approach for quasiquoted code that references symbols in the lexical scope of where the quasiquoted code appears.) This seemed promising, but there’s a problem:
my $x = BEGIN %*ENV<DEBUG> ?? -> $x { note "Got $x"; foo($x) } !! -> $x { foo($x) };It’s fine to post-declare subs, and so there’s no value to fixate. Thankfully, the generalized dispatch mechanism can ride to the rescue; we can:
When compiling Raku code, timing is everything. I knew this and tried to account for it in the RakuAST design from the start, but a couple of things in particular turned out a bit awkward.
I got a decent way into this restructuring work during the core summit, and hope to find time soon to get it a bit further along (I’ve been a mix of busy, tired, and had an eye infection to boot since getting back from the summit, so thus far there’s not been time for it).
I also took part in various other discussions and helped with some other things; those that are probably most worth mentioning are:
Thanks goes to Liz for organizing the summit, to Wendy for keeping everyone so well fed and watered, to the rest of attendees for many interesting discussions over the three days, to TPRF and Rootprompt for sponsoring the event, and to Edument for supporting my attendance.
Hi hackers! Today the MoarVM JIT project is nearly 9 years old. I was inspired by Jonathan's presentation reflecting on the development of MoarVM, to do the same for the MoarVM JIT, for which I have been responsible.
For those who are unfamiliar, what is commonly understood as 'JIT compilation' for virtual machines is performed by two components in MoarVM.
This post refers only to the native code generation backend component. It, too, is split into two mostly-independent systems:
One one hand, as a result of my limited experience, time and resources, and on the other hand as a result of the design of MoarVM.
MoarVM was originally designed as a traditional interpreter for a high level language (much like the Perl interpreter). Meaning that it has a large number of different instructions and many instructions operate on high-level data structures like strings, arrays and maps (as opposed to pointers and machine words).
This is by no means a bad or outdated design. Frequently executed routines (string manipulation, hash table lookups etc.) are implemented using an efficient language (C) and driven by a language that is optimized for usability (Raku). This design is also used in modern machine learning frameworks. More importantly, this was a reasonable design because it is a good target for the Rakudo compiler.
For the JIT compiler, this means two things:
The machine code generated by the JIT compiler then will mostly consists of consecutive function calls to VM routines, which is not the type of code where a compiler can really improve performance much.
In other words, suppose 50% of runtime is spent in interpretation overhead (instruction decoding and dispatch), and 50% is spent in VM routines, then removing interpretation overhead via JIT compilation will at best result in a twofold increase in performance. For many programs, the observed performance increase will be even less.
Mind that I'm specifically refering to the improvement due to machine code generation, and not to those due to type specialization, inlining etc. (the domain of 'spesh'). These latter features have resulted in much more significant performance improvements.
For me personally, it was a tremendously valuable learning experience which led directly to my current career, writing SQL compilers for Google Cloud.
For the Raku community, even if we never realized the performance improvements that I might have hoped at the start, I hope that the JIT project (as it exists) has been valuable, if for no other reason than identifying the challenges of JIT compilation for MoarVM. A future effort may be able to do better based on what we learned; and I hope my blog posts are a useful resource from that perspective.
Assuming that time and resources were not an issue:
If any of this comes to pass, you'll find my report on it right here. Thanks for reasding and until then!
Around 18 months ago, I set about working on the largest set of architectural changes that Raku runtime MoarVM has seen since its inception. The work was most directly triggered by the realization that we had no good way to fix a certain semantic bug in dispatch without either causing huge performance impacts across the board or increasingly complexity even further in optimizations that were already riding their luck. However, the need for something like this had been apparent for a while: a persistent struggle to optimize certain Raku language features, the pain of a bunch of performance mechanisms that were all solving the same kind of problem but each for a specific situation, and a sense that, with everything learned since I founded MoarVM, it was possible to do better.
The result is the development of a new generalized dispatch mechanism. An overview can be found in my Raku Conference talk about it (slides, video); in short, it gives us a far more uniform architecture for all kinds of dispatch, allowing us to deliver better performance on a range of language features that have thus far been glacial, as well as opening up opportunities for new optimizations.
Today, this work has been merged, along with the matching changes in NQP (the Raku subset we use for bootstrapping and to implement the compiler) and Rakudo (the full Raku compiler and standard library implementation). This means that it will ship in the October 2021 releases.
In this post, I’ll give an overview of what you can expect to observe right away, and what you might expect in the future as we continue to build upon the possibilities that the new dispatch architecture has to offer.
The biggest improvements involve language features that we’d really not had the architecture to do better on before. They involved dispatch – that is, getting a call linked to a destination efficiently – but the runtime didn’t provide us with a way to “explain” to it that it was looking at a dispatch, let alone with the information needed to have a shot at optimizing it.
The following graph captures a number of these cases, and shows the level of improvement, ranging from a factor of 3.3 to 13.3 times faster.
Let’s take a quick look at each of these. The first, new-buf, asks how quickly we can allocate Bufs.
for ^10_000_000 {
Buf.new
}
Why is this a dispatch benchmark? Because Buf is not a class, but rather a role. When we try to make an instance of a role, it is “punned” into a class. Up until now, it works as follows:
new methodfind_method method would, if needed, create a pun of the role and cache it-> $role-discarded, |args { $pun."$name"(|args) }This had a number of undesirable consequences:
With the new dispatch mechanism, we have a means to cache constants at a given program location and to replace arguments. So the first time we encounter the call, we:
new method on the class punned from the roleFor the next thousands of calls, we interpret this dispatch program. It’s still some cost, but the method we’re calling is already resolved, and the argument list rewriting is fairly cheap. Meanwhile, after we get into some hundreds of iterations, on a background thread, the optimizer gets to work. The argument re-ordering cost goes away completely at this point, and new is so small it gets inlined – at which point the buffer allocation is determined dead and so goes away too. Some remaining missed opportunities mean we still are left with a loop that’s not quite empty: it busies itself making sure it’s really OK to do nothing, rather than just doing nothing.
Next up, multiple dispatch with where clauses.
multi fac($n where $n <= 1) { 1 }
multi fac($n) { $n * fac($n - 1) }
for ^1_000_000 {
fac(5)
}
These were really slow before, since:
where clause involvedwhere clauses twice in the event the candidate was chosen: once to see if we should choose that multi candidate, and once again when we entered itWith the new mechanism, we:
where clause, in a mode whereby if the signature fails to bind, it triggers a dispatch resumption. (If it does bind, it runs to completion)Once again, after the setup phase, we interpret the dispatch programs. In fact, that’s as far as we get with running this faster for now, because the specializer doesn’t yet know how to translate and further optimize this kind of dispatch program. (That’s how I know it currently stands no chance of turning this whole thing into another empty loop!) So there’s more to be had here also; in the meantime, I’m afraid you’ll just have to settle for a factor of ten speedup.
Here’s the next one:
proto with-proto(Int $n) { 2 * {*} }
multi with-proto(Int $n) { $n + 1 }
sub invoking-nontrivial-proto() {
for ^10_000_000 {
with-proto(20)
}
}
Again, on top form, we’d turn this into an empty loop too, but we don’t quite get there yet. This case wasn’t so terrible before: we did get to use the multiple dispatch cache, however to do that we also ended up having to allocate an argument capture. The need for this also blocked any chance of inlining the proto into the caller. Now that is possible. Since we cannot yet translate dispatch programs that resume an in-progress dispatch, we don’t yet get to further inline the called multi candidate into the proto. However, we now have a design that will let us implement that.
This whole notion of a dispatch resumption – where we start doing a dispatch, and later need to access arguments or other pre-calculated data in order to do a next step of it – has turned out to be a great unification. The initial idea for it came from considering things like callsame:
class Parent {
method m() { 1 }
}
class Child is Parent {
method m() { 1 + callsame }
}
for ^10_000_000 {
Child.m;
}
Once I started looking at this, and then considering that a complex proto also wants to continue with a dispatch at the {*}, and in the case a where clauses fails in a multi it also wants to continue with a dispatch, I realized this was going to be useful for quite a lot of things. It will be a bit of a headache to teach the optimizer and JIT to do nice things with resumes – but a great relief that doing that once will benefit multiple language features!
Anyway, back to the benchmark. This is another “if we were smart, it’d be an empty loop” one. Previously, callsame was very costly, because each time we invoked it, it would have to calculate what kind of dispatch we were resuming and the set of methods to call. We also had to be able to locate the arguments. Dynamic variables were involved, which cost a bit to look up too, and – despite being an implementation details – these also leaked out in introspection, which wasn’t ideal. The new dispatch mechanism makes this all rather more efficient: we can cache the calculated set of methods (or wrappers and multi candidates, depending on the context) and then walk through it, and there’s no dynamic variables involved (and thus no leakage of them). This sees the biggest speedup of the lot – and since we cannot yet inline away the callsame, it’s (for now) measuring the speedup one might expect on using this language feature. In the future, it’s destined to optimize away to an empty loop.
A module that makes use of callsame on a relatively hot path is OO::Monitors,, so I figured it would be interesting to see if there is a speedup there also.
use OO::Monitors;
monitor TestMonitor {
method m() { 1 }
}
my $mon = TestMonitor.new;
for ^1_000_000 {
$mon.m();
}
A monitor is a class that acquires a lock around each method call. The module provides a custom meta-class that adds a lock attribute to the class and then wraps each method such that it acquires the lock. There are certainly costly things in there besides the involvement of callsame, but the improvement to callsame is already enough to see a 3.3x speedup in this benchmark. Since OO::Monitors is used in quite a few applications and modules (for example, Cro uses it), this is welcome (and yes, a larger improvement will be possible here too).
I’ve seen some less impressive, but still welcome, improvements across a good number of other microbenchmarks. Even a basic multi dispatch on the + op:
my $i = 0;
for ^10_000_000 {
$i = $i + $_;
}
Comes out with a factor of 1.6x speedup, thanks primarily to us producing far tighter code with fewer guards. Previously, we ended up with duplicate guards in this seemingly straightforward case. The infix:<+> multi candidate would be specialized for the case of its first argument being an Int in a Scalar container and its second argument being an immutable Int. Since a Scalar is mutable, the specialization would need to read it and then guard the value read before proceeding, otherwise it may change, and we’d risk memory safety. When we wanted to inline this candidate, we’d also want to do a check that the candidate really applies, and so also would deference the Scalar and guard its content to do that. We can and do eliminate duplicate guards – but these guards are on two distinct reads of the value, so that wouldn’t help.
Since in the new dispatch mechanism we can rewrite arguments, we can now quite easily do caller-side removal of Scalar containers around values. So easily, in fact, that the change to do it took me just a couple of hours. This gives a lot of benefits. Since dispatch programs automatically eliminate duplicate reads and guards, the read and guard by the multi-dispatcher and the read in order to pass the decontainerized value are coalesced. This means less repeated work prior to specialization and JIT compilation, and also only a single read and guard in the specialized code after it. With the value to be passed already guarded, we can trivially select a candidate taking two bare Int values, which means there’s no further reads and guards needed in the callee either.
A less obvious benefit, but one that will become important with planned future work, is that this means Scalar containers escape to callees far less often. This creates further opportunities for escape analysis. While the MoarVM escape analyzer and scalar replacer is currently quite limited, I hope to return to working on it in the near future, and expect it will be able to give us even more value now than it would have been able to before.
The benchmarks shown earlier are mostly of the “how close are we to realizing that we’ve got an empty loop” nature, which is interesting for assessing how well the optimizer can “see through” dispatches. Here are a few further results on more “traditional” microbenchmarks:
The complex number benchmark is as follows:
my $total-re = 0e0;
for ^2_000_000 {
my $x = 5 + 2i;
my $y = 10 + 3i;
my $z = $x * $x + $y;
$total-re = $total-re + $z.re
}
say $total-re;
That is, just a bunch of operators (multi dispatch) and method calls, where we really do use the result. For now, we’re tied with Python and a little behind Ruby on this benchmark (and a surprising 48 times faster than the same thing done with Perl’s Math::Complex), but this is also a case that stands to see a huge benefit from escape analysis and scalar replacement in the future.
The hash read benchmark is:
my %h = a => 10, b => 12;
my $total = 0;
for ^10_000_000 {
$total = $total + %h<a> + %h<b>;
}
And the hash store one is:
my @keys = 'a'..'z';
for ^500_000 {
my %h;
for @keys {
%h{$_} = 42;
}
}
The improvements are nothing whatsoever to do with hashing itself, but instead look to be mostly thanks to much tighter code all around due to caller-side decontainerization. That can have a secondary effect of bringing things under the size limit for inlining, which is also a big help. Speedup factors of 2x and 1.85x are welcome, although we could really do with the same level of improvement again for me to be reasonably happy with our results.
The line-reading benchmark is:
my $fh = open "longfile";
my $chars = 0;
for $fh.lines { $chars = $chars + .chars };
$fh.close;
say $chars
Again, nothing specific to I/O got faster, but when dispatch – the glue that puts together all the pieces – gets a boost, it helps all over the place. (We are also decently competitive on this benchmark, although tend to be slower the moment the UTF-8 decoder can’t take it’s “NFG can’t possibly apply” fast path.)
I’ve also started looking at larger programs, and hearing results from others about theirs. It’s mostly encouraging:
Text::CSV benchmark test-t has seen roughly 20% improvement (thanks to lizmat for measuring)Cro::HTTP test application gets through about 10% more requests per secondCORE.setting, the standard library. However, a big pinch of salt is needed here: the compiler itself has changed in a number of places as part of the work, and there were a couple of things tweaked based on looking at profiles that aren’t really related to dispatch.One unpredicted (by me), but also welcome, improvement is that profiler output has become significantly smaller. Likely reasons for this include:
sink method when a value was in sink context. Now, if we see that the type simply inherits that method from Mu, we elide the call entirely (again, it would inline away, but a smaller call graph is a smaller profile).proto when the cache was missed, but would then not call an onlystar proto again when it got cache hits in the future. This meant the call tree under many multiple dispatches was duplicated in the profile. This wasn’t just a size issue; it was a bit annoying to have this effect show up in the profile reports too.To give an example of the difference, I took profiles from Agrammon to study why it might have become slower. The one from before the dispatcher work weighed in at 87MB; the one with the new dispatch mechanism is under 30MB. That means less memory used while profiling, less time to write the profile out to disk afterwards, and less time for tools to load the profiler output. So now it’s faster to work out how to make things faster.
I’m afraid so. Startup time has suffered. While the new dispatch mechanism is more powerful, pushes more complexity out of the VM into high level code, and is more conducive to reaching higher peak performance, it also has a higher warmup time. At the time of writing, the impact on startup time seems to be around 25%. I expect we can claw some of that back ahead of the October release.
Changes of this scale always come with an amount of risk. We’re merging this some weeks ahead of the next scheduled monthly release in order to have time for more testing, and to address any regressions that get reported. However, even before reaching the point of merging it, we have:
blin to run the tests of ecosystem modules. This is a standard step when preparing Rakudo releases, but in this case we’ve aimed it at the new-disp branches. This found a number of regressions caused by the switch to the new dispatch mechanism, which have been addressed.As I’ve alluded to in a number of places in this post, while there are improvements to be enjoyed right away, there are also new opportunities for further improvement. Some things that are on my mind include:
callsame one here is a perfect example! The point we do the resumption of a dispatch is inside callsame, so all the inline cache entries of resumptions throughout the program stack up in one place. What we’d like is to have them attached a level down the callstack instead. Otherwise, the level of callsame improvement seen in micro-benchmarks will not be enjoyed in larger applications. This applies in a number of other situations too.FALLBACK method could have its callsite easily rewritten to do that, opening the way to inlining.Ints (which needs a great deal of care in memory management, as they may box a big integer, not just a native integer).I would like to thank TPF and their donors for providing the funding that has made it possible for me to spend a good amount of my working time on this effort.
While I’m to blame for the overall design and much of the implementation of the new dispatch mechanism, plenty of work has also been put in by other MoarVM and Rakudo contributors – especially over the last few months as the final pieces fell into place, and we turned our attention to getting it production ready. I’m thankful to them not only for the code and debugging contributions, but also much support and encouragement along the way. It feels good to have this merged, and I look forward to building upon it in the months and years to come.
I recently wrote about the new MoarVM dispatch mechanism, and in that post noted that I still had a good bit of Raku’s multiple dispatch semantics left to implement in terms of it. Since then, I’ve made a decent amount of progress in that direction. This post contains an overview of the approach taken, and some very rough performance measurements.
Of all the kinds of dispatch we find in Raku, multiple dispatch is the most complex. Multiple dispatch allows us to write a set of candidates, which are then selected by the number of arguments:
multi ok($condition, $desc) {
say ($condition ?? 'ok' !! 'not ok') ~ " - $desc";
}
multi ok($condition) {
ok($condition, '');
}
Or the types of arguments:
multi to-json(Int $i) { ~$i }
multi to-json(Bool $b) { $b ?? 'true' !! 'false' }
And not just one argument, but potentially many:
multi truncate(Str $str, Int $chars) {
$str.chars < $chars ?? $str !! $str.substr(0, $chars) ~ '...'
}
multi truncate(Str $str, Str $after) {
with $str.index($after) -> $pos {
$str.substr(0, $pos) ~ '...'
}
else {
$str
}
}
We may write where clauses to differentiate candidates on properties that are not captured by nominal types:
multi fac($n where $n <= 1) { 1 }
multi fac($n) { $n * fac($n - 1) }
Every time we write a set of multi candidates like this, the compiler will automatically produce a proto routine. This is what is installed in the symbol table, and holds the candidate list. However, we can also write our own proto, and use the special term {*} to decide at which point we do the dispatch, if at all.
proto mean($collection) {
$collection.elems == 0 ?? Nil !! {*}
}
multi mean(@arr) {
@arr.sum / @arr.elems
}
multi mean(%hash) {
%hash.values.sum / %hash.elems
}
Candidates are ranked by narrowness (using topological sorting). If multiple candidates match, but they are equally narrow, then that’s an ambiguity error. Otherwise, we call narrowest one. The candidate we choose may then use callsame and friends to defer to the next narrowest candidate, which may do the same, until we reach the most general matching one.
Raku leans heavily on multiple dispatch. Most operators in Raku are compiled into calls to multiple dispatch subroutines. Even $a + $b will be a multiple dispatch. This means doing multiple dispatch efficiently is really important for performance. Given the riches of its semantics, this is potentially a bit concerning. However, there’s good news too.
The overwhelmingly common case is that we have:
where clausesprotocallsameThis isn’t to say the other cases are unimportant; they are really quite useful, and it’s desirable for them to perform well. However, it’s also desirable to make what savings we can in the common case. For example, we don’t want to eagerly calculate the full set of possible candidates for every single multiple dispatch, because the majority of the time only the first one matters. This is not just a time concern: recall that the new dispatch mechanism stores dispatch programs at each callsite, and if we store the list of all matching candidates at each of those, we’ll waste a lot of memory too.
The situation in Rakudo today is as follows:
proto holding a “dispatch cache”, a special-case mechanism implemented in the VM that uses a search tree, with one level per argument.proto, it’s not too bad either, though inlining isn’t going to be happening; it can still use the search tree, thoughwhere clauses, it’ll be slow, because the search tree only deals in finding one candidate per set of nominal types, and so we can’t use itcallsame; it’ll be slow tooEffectively, the situation today is that you simply don’t use where clauses in a multiple dispatch if its anywhere near a hot path (well, and if you know where the hot paths are, and know that this kind of dispatch is slow). Ditto for callsame, although that’s less commonly reached for. The question is, can we do better with the new dispatcher?
Let’s start out with seeing how the simplest cases are dealt with, and build from there. (This is actually what I did in terms of the implementation, but at the same time I had a rough idea where I was hoping to end up.)
Recall this pair of candidates:
multi truncate(Str $str, Int $chars) {
$str.chars < $chars ?? $str !! $str.substr(0, $chars) ~ '...'
}
multi truncate(Str $str, Str $after) {
with $str.index($after) -> $pos {
$str.substr(0, $pos) ~ '...'
}
else {
$str
}
}
We then have a call truncate($message, "\n"), where $message is a Str. Under the new dispatch mechanism, the call is made using the raku-call dispatcher, which identifies that this is a multiple dispatch, and thus delegates to raku-multi. (Multi-method dispatch ends up there too.)
The record phase of the dispatch – on the first time we reach this callsite – will proceed as follows:
raku-invoke dispatcher with the chosen candidate.When we reach the same callsite again, we can run the dispatch program, which quickly checks if the argument types match those we saw last time, and if they do, we know which candidate to invoke. These checks are very cheap – far cheaper than walking through all of the candidates and examining each of them for a match. The optimizer may later be able to prove that the checks will always come out true and eliminate them.
Thus the whole of the dispatch processes – at least for this simple case where we only have types and arity – can be “explained” to the virtual machine as “if the arguments have these exact types, invoke this routine”. It’s pretty much the same as we were doing for method dispatch, except there we only cared about the type of the first argument – the invocant – and the value of the method name. (Also recall from the previous post that if it’s a multi-method dispatch, then both method dispatch and multiple dispatch will guard the type of the first argument, but the duplication is eliminated, so only one check is done.)
Coming up with good abstractions is difficult, and therein lies much of the challenge of the new dispatch mechanism. Raku has quite a number of different dispatch-like things. However, encoding all of them directly in the virtual machine leads to high complexity, which makes building reliable optimizations (or even reliable unoptimized implementations!) challenging. Thus the aim is to work out a comparatively small set of primitives that allow for dispatches to be “explained” to the virtual machine in such a way that it can deliver decent performance.
It’s fairly clear that callsame is a kind of dispatch resumption, but what about the custom proto case and the where clause case? It turns out that these can both be neatly expressed in terms of dispatch resumption too (the where clause case needing one small addition at the virtual machine level, which in time is likely to be useful for other things too). Not only that, but encoding these features in terms of dispatch resumption is also quite direct, and thus should be efficient. Every trick we teach the specializer about doing better with dispatch resumptions can benefit all of the language features that are implemented using them, too.
Recall this example:
proto mean($collection) {
$collection.elems == 0 ?? Nil !! {*}
}
Here, we want to run the body of the proto, and then proceed to the chosen candidate at the point of the {*}. By contrast, when we don’t have a custom proto, we’d like to simply get on with calling the correct multi.
To achieve this, I first moved the multi candidate selection logic from the raku-multi dispatcher to the raku-multi-core dispatcher. The raku-multi dispatcher then checks if we have an “onlystar” proto (one that does not need us to run it). If so, it delegates immediately to raku-multi-core. If not, it saves the arguments to the dispatch as the resumption initialization state, and then calls the proto. The proto‘s {*} is compiled into a dispatch resumption. The resumption then delegates to raku-multi-core. Or, in code:
nqp::dispatch('boot-syscall', 'dispatcher-register', 'raku-multi',
# Initial dispatch, only setting up resumption if we need to invoke the
# proto.
-> $capture {
my $callee := nqp::captureposarg($capture, 0);
my int $onlystar := nqp::getattr_i($callee, Routine, '$!onlystar');
if $onlystar {
# Don't need to invoke the proto itself, so just get on with the
# candidate dispatch.
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-multi-core', $capture);
}
else {
# Set resume init args and run the proto.
nqp::dispatch('boot-syscall', 'dispatcher-set-resume-init-args', $capture);
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-invoke', $capture);
}
},
# Resumption means that we have reached the {*} in the proto and so now
# should go ahead and do the dispatch. Make sure we only do this if we
# are signalled to that it's a resume for an onlystar (resumption kind 5).
-> $capture {
my $track_kind := nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $capture, 0);
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal', $track_kind);
my int $kind := nqp::captureposarg_i($capture, 0);
if $kind == 5 {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-multi-core',
nqp::dispatch('boot-syscall', 'dispatcher-get-resume-init-args'));
}
elsif !nqp::dispatch('boot-syscall', 'dispatcher-next-resumption') {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-constant',
nqp::dispatch('boot-syscall', 'dispatcher-insert-arg-literal-obj',
$capture, 0, Nil));
}
});
Deferring to the next candidate (for example with callsame) and trying the next candidate because a where clause failed look very similar: both involve walking through a list of possible candidates. There’s some details, but they have a great deal in common, and it’d be nice if that could be reflected in how multiple dispatch is implemented using the new dispatcher.
Before that, a slightly terrible detail about how things work in Rakudo today when we have where clauses. First, the dispatcher does a “trial bind”, where it asks the question: would this signature bind? To do this, it has to evaluate all of the where clauses. Worse, it has to use the slow-path signature binder too, which interprets the signature, even though we can in many cases compile it. If the candidate matches, great, we select it, and then invoke it…which runs the where clauses a second time, as part of the compiled signature binding code. There is nothing efficient about this at all, except for it being by far more efficient on developer time, which is why it happened that way.
Anyway, it goes without saying that I’m rather keen to avoid this duplicate work and the slow-path binder where possible as I re-implement this using the new dispatcher. And, happily, a small addition provides a solution. There is an op assertparamcheck, which any kind of parameter checking compiles into (be it type checking, where clause checking, etc.) This triggers a call to a function that gets the arguments, the thing we were trying to call, and can then pick through them to produce an error message. The trick is to provide a way to invoke a routine such that a bind failure, instead of calling the error reporting function, will leave the routine and then do a dispatch resumption! This means we can turn failure to pass where clause checks into a dispatch resumption, which will then walk to the next candidate and try it instead.
This gets us most of the way to a solution, but there’s still the question of being memory and time efficient in the common case, where there is no resumption and no where clauses. I coined the term “trivial multiple dispatch” for this situation, which makes the other situation “non-trivial”. In fact, I even made a dispatcher called raku-multi-non-trivial! There are two ways we can end up there.
where clauses. As soon as we see this is the case, we go ahead and produce a full list of possible candidates that could match. This is a linked list (see my previous post for why).callsame happens, we end up in the trivial dispatch resumption handler, which – since this situation is now non-trivial – builds the full candidate list, snips the first item off it (because we already ran that), and delegates to raku-multi-non-trivial.Lost in this description is another significant improvement: today, when there are where clauses, we entirely lose the ability to use the MoarVM multiple dispatch cache, but under the new dispatcher, we store a type-filtered list of candidates at the callsite, and then cheap type guards are used to check it is valid to use.
I did a few benchmarks to see how the new dispatch mechanism did with a couple of situations known to be sub-optimal in Rakudo today. These numbers do not reflect what is possible, because at the moment the specializer does not have much of an understanding of the new dispatcher. Rather, they reflect the minimal improvement we can expect.
Consider this benchmark using a multi with a where clause to recursively implement factorial.
multi fac($n where $n <= 1) { 1 }
multi fac($n) { $n * fac($n - 1) }
for ^100_000 {
fac(10)
}
say now - INIT now;
This needs some tweaks (and to be run under an environment variable) to use the new dispatcher; these are temporary, until such a time I switch Rakudo over to using the new dispatcher by default:
use nqp;
multi fac($n where $n <= 1) { 1 }
multi fac($n) { $n * nqp::dispatch('raku-call', &fac, $n - 1) }
for ^100_000 {
nqp::dispatch('raku-call', &fac, 10);
}
say now - INIT now;
On my machine, the first runs in 4.86s, the second in 1.34s. Thus under the new dispatcher this runs in little over a quarter of the time it used to – a quite significant improvement already.
A case involving callsame is also interesting to consider. Here it is without using the new dispatcher:
multi fallback(Any $x) { "a$x" }
multi fallback(Numeric $x) { "n" ~ callsame }
multi fallback(Real $x) { "r" ~ callsame }
multi fallback(Int $x) { "i" ~ callsame }
for ^1_000_000 {
fallback(4+2i);
fallback(4.2);
fallback(42);
}
say now - INIT now;
And with the temporary tweaks to use the new dispatcher:
use nqp;
multi fallback(Any $x) { "a$x" }
multi fallback(Numeric $x) { "n" ~ new-disp-callsame }
multi fallback(Real $x) { "r" ~ new-disp-callsame }
multi fallback(Int $x) { "i" ~ new-disp-callsame }
for ^1_000_000 {
nqp::dispatch('raku-call', &fallback, 4+2i);
nqp::dispatch('raku-call', &fallback, 4.2);
nqp::dispatch('raku-call', &fallback, 42);
}
say now - INIT now;
On my machine, the first runs in 31.3s, the second in 11.5s, meaning that with the new dispatcher we manage it in a little over a third of the time that current Rakudo does.
These are both quite encouraging, but as previously mentioned, a majority of multiple dispatches are of the trivial kind, not using these features. If I make the most common case worse on the way to making other things better, that would be bad. It’s not yet possible to make a fair comparison of this: trivial multiple dispatches already receive a lot of attention in the specializer, and it doesn’t yet optimize code using the new dispatcher well. Of note, in an example like this:
multi m(Int) { }
multi m(Str) { }
for ^1_000_000 {
m(1);
m("x");
}
say now - INIT now;
Inlining and other optimizations will turn this into an empty loop, which is hard to beat. There is one thing we can already do, though: run it with the specializer disabled. The new dispatcher version looks like this:
use nqp;
multi m(Int) { }
multi m(Str) { }
for ^1_000_000 {
nqp::dispatch('raku-call', &m, 1);
nqp::dispatch('raku-call', &m, "x");
}
say now - INIT now;
The results are 0.463s and 0.332s respectively. Thus, the baseline execution time – before the specializer does its magic – is less using the new general dispatch mechanism than it is using the special-case multiple dispatch cache that we currently use. I wasn’t sure what to expect here before I did the measurement. Given we’re going from a specialized mechanism that has been profiled and tweaked to a new general mechanism that hasn’t received such attention, I was quite ready to be doing a little bit worse initially, and would have been happy with parity. Running in 70% of the time was a bigger improvement than I expected at this point.
I expect that once the specializer understands the new dispatch mechanism better, it will be able to also turn the above into an empty loop – however, since more iterations can be done per-optimization, this should still show up as a win for the new dispatcher.
With one relatively small addition, the new dispatch mechanism is already handling most of the Raku multiple dispatch semantics. Furthermore, even without the specializer and JIT really being able to make a good job of it, some microbenchmarks already show a factor of 3x-4x improvement. That’s a pretty good starting point.
There’s still a good bit to do before we ship a Rakudo release using the new dispatcher. However, multiple dispatch was the biggest remaining threat to the design: it’s rather more involved than other kinds of dispatch, and it was quite possible that an unexpected shortcoming could trigger another round of design work, or reveal that the general mechanism was going to struggle to perform compared to the more specialized one in the baseline unoptimized, case. So far, there’s no indication of either of these, and I’m cautiously optimistic that the overall design is about right.
My goodness, it appears I’m writing my first Raku internals blog post in over two years. Of course, two years ago it wasn’t even called Raku. Anyway, without further ado, let’s get on with this shared brainache.
I use “dispatch” to mean a process by which we take a set of arguments and end up with some action being taken based upon them. Some familiar examples include:
$basket.add($product, $quantity). We might traditionally call just $product and $qauntity the arguments, but for my purposes, all of $basket, the method name 'add', $product, and $quantity` are arguments to the dispatch: they are the things we need in order to make a decision about what we’re going to do.uc($youtube-comment). Since Raku sub calls are lexically resolved, in this case the arguments to the dispatch are &uc (the result of looking up the subroutine) and $youtube-comment.At first glance, perhaps the first two seem fairly easy and the third a bit more of a handful – which is sort of true. However, Raku has a number of other features that make dispatch rather more, well, interesting. For example:
wrap allows us to wrap any Routine (sub or method); the wrapper can then choose to defer to the original routine, either with the original arguments or with new argumentsproto routine that gets to choose when – or even if – the call to the appropriate candidate is madecallsame in order to defer to the next candidate in the dispatch. But what does that mean? If we’re in a multiple dispatch, it would mean the next most applicable candidate, if any. If we’re in a method dispatch then it means a method from a base class. (The same thing is used to implement going to the next wrapper or, eventually, to the originally wrapped routine too). And these can be combined: we can wrap a multi method, meaning we can have 3 levels of things that all potentially contribute the next thing to call!Thanks to this, dispatch – at least in Raku – is not always something we do and produce an outcome, but rather a process that we may be asked to continue with multiple times!
Finally, while the examples I’ve written above can all quite clearly be seen as examples of dispatch, a number of other common constructs in Raku can be expressed as a kind of dispatch too. Assignment is one example: the semantics of it depend on the target of the assignment and the value being assigned, and thus we need to pick the correct semantics. Coercion is another example, and return value type-checking yet another.
Dispatch is everywhere in our programs, quietly tieing together the code that wants stuff done with the code that does stuff. Its ubiquity means it plays a significant role in program performance. In the best case, we can reduce the cost to zero. In the worst case, the cost of the dispatch is high enough to exceed that of the work done as a result of the dispatch.
To a first approximation, when the runtime “understands” the dispatch the performance tends to be at least somewhat decent, but when it doesn’t there’s a high chance of it being awful. Dispatches tend to involve an amount of work that can be cached, often with some cheap guards to verify the validity of the cached outcome. For example, in a method dispatch, naively we need to walk a linearization of the inheritance graph and ask each class we encounter along the way if it has a method of the specified name. Clearly, this is not going to be terribly fast if we do it on every method call. However, a particular method name on a particular type (identified precisely, without regard to subclassing) will resolve to the same method each time. Thus, we can cache the outcome of the lookup, and use it whenever the type of the invocant matches that used to produce the cached result.
When one starts building a runtime aimed at a particular language, and has to do it on a pretty tight budget, the most obvious way to get somewhat tolerable performance is to bake various hot-path language semantics into the runtime. This is exactly how MoarVM started out. Thus, if we look at MoarVM as it stood several years ago, we find things like:
where comes at a very high cost)Sub object has a private attribute in it that holds the low-level code handle identifying the bytecode to run)These are all still there today, however are also all on the way out. What’s most telling about this list is what isn’t included. Things like:
$obj.SomeType::method-name())A few years back I started to partially address this, with the introduction of a mechanism I called “specializer plugins”. But first, what is the specializer?
When MoarVM started out, it was a relatively straightforward interpreter of bytecode. It only had to be fast enough to beat the Parrot VM in order to get a decent amount of usage, which I saw as important to have before going on to implement some more interesting optimizations (back then we didn’t have the kind of pre-release automated testing infrastructure we have today, and so depended much more on feedback from early adopters). Anyway, soon after being able to run pretty much as much of the Raku language as any other backend, I started on the dynamic optimizer. It gathered type statistics as the program was interpreted, identified hot code, put it into SSA form, used the type statistics to insert guards, used those together with static properties of the bytecode to analyze and optimize, and produced specialized bytecode for the function in question. This bytecode could elide type checks and various lookups, as well as using a range of internal ops that make all kinds of assumptions, which were safe because of the program properties that were proved by the optimizer. This is called specialized bytecode because it has had a lot of its genericity – which would allow it to work correctly on all types of value that we might encounter – removed, in favor of working in a particular special case that actually occurs at runtime. (Code, especially in more dynamic languages, is generally far more generic in theory than it ever turns out to be in practice.)
This component – the specializer, known internally as “spesh” – delivered a significant further improvement in the performance of Raku programs, and with time its sophistication has grown, taking in optimizations such as inlining and escape analysis with scalar replacement. These aren’t easy things to build – but once a runtime has them, they create design possibilities that didn’t previously exist, and make decisions made in their absence look sub-optimal.
Of note, those special-cased language-specific mechanisms, baked into the runtime to get some speed in the early days, instead become something of a liability and a bottleneck. They have complex semantics, which means they are either opaque to the optimizer (so it can’t reason about them, meaning optimization is inhibited) or they need special casing in the optimizer (a liability).
So, back to specializer plugins. I reached a point where I wanted to take on the performance of things like $obj.?meth (the “call me maybe” dispatch), $obj.SomeType::meth() (dispatch qualified with a class to start looking in), and private method calls in roles (which can’t be resolved statically). At the same time, I was getting ready to implement some amount of escape analysis, but realized that it was going to be of very limited utility because assignment had also been special-cased in the VM, with a chunk of opaque C code doing the hot path stuff.
But why did we have the C code doing that hot-path stuff? Well, because it’d be too espensive to have every assignment call a VM-level function that does a bunch of checks and logic. Why is that costly? Because of function call overhead and the costs of interpretation. This was all true once upon a time. But, some years of development later:
I solved the assignment problem and the dispatch problems mentioned above with the introduction of a single new mechanism: specializer plugins. They work as follows:
The vast majority of cases are monomorphic, meaning that only one set of guards are produced and they always succeed thereafter. The specializer can thus compile those guards into the specialized bytecode and then assume the given target invocant is what will be invoked. (Further, duplicate guards can be eliminated, so the guards a particular plugin introduces may reduce to zero.)
Specializer plugins felt pretty great. One new mechanism solved multiple optimization headaches.
The new MoarVM dispatch mechanism is the answer to a fairly simple question: what if we get rid of all the dispatch-related special-case mechanisms in favor of something a bit like specializer plugins? The resulting mechanism would need to be a more powerful than specializer plugins. Further, I could learn from some of the shortcomings of specializer plugins. Thus, while they will go away after a relatively short lifetime, I think it’s fair to say that I would not have been in a place to design the new MoarVM dispatch mechanism without that experience.
All the method caching. All the multi dispatch caching. All the specializer plugins. All the invocation protocol stuff for unwrapping the bytecode handle in a code object. It’s all going away, in favor of a single new dispatch instruction. Its name is, boringly enough, dispatch. It looks like this:
dispatch_o result, 'dispatcher-name', callsite, arg0, arg1, ..., argN
Which means:
dispatcher-nameresult(Aside: this implies a new calling convention, whereby we no longer copy the arguments into an argument buffer, but instead pass the base of the register set and a pointer into the bytecode where the register argument map is found, and then do a lookup registers[map[argument_index]] to get the value for an argument. That alone is a saving when we interpret, because we no longer need a loop around the interpreter per argument.)
Some of the arguments might be things we’d traditionally call arguments. Some are aimed at the dispatch process itself. It doesn’t really matter – but it is more optimal if we arrange to put arguments that are only for the dispatch first (for example, the method name), and those for the target of the dispatch afterwards (for example, the method parameters).
The new bootstrap mechanism provides a small number of built-in dispatchers, whose names start with “boot-“. They are:
boot-value– take the first argument and use it as the result (the identity function, except discarding any further arguments)boot-constant – take the first argument and produce it as the result, but also treat it as a constant value that will always be produced (thus meaning the optimizer could consider any pure code used to calculate the value as dead)boot-code – take the first argument, which must be a VM bytecode handle, and run that bytecode, passing the rest of the arguments as its parameters; evaluate to the return value of the bytecodeboot-syscall – treat the first argument as the name of a VM-provided built-in operation, and call it, providing the remaining arguments as its parametersboot-resume – resume the topmost ongoing dispatchThat’s pretty much it. Every dispatcher we build, to teach the runtime about some other kind of dispatch behavior, eventually terminates in one of these.
Teaching MoarVM about different kinds of dispatch is done using nothing less than the dispatch mechanism itself! For the most part, boot-syscall is used in order to register a dispatcher, set up the guards, and provide the result that goes with them.
Here is a minimal example, taken from the dispatcher test suite, showing how a dispatcher that provides the identity function would look:
nqp::dispatch('boot-syscall', 'dispatcher-register', 'identity', -> $capture {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-value', $capture);
});
sub identity($x) {
nqp::dispatch('identity', $x)
}
ok(identity(42) == 42, 'Can define identity dispatch (1)');
ok(identity('foo') eq 'foo', 'Can define identity dispatch (2)');
In the first statement, we call the dispatcher-register MoarVM system call, passing a name for the dispatcher along with a closure, which will be called each time we need to handle the dispatch (which I tend to refer to as the “dispatch callback”). It receives a single argument, which is a capture of arguments (not actually a Raku-level Capture, but the idea – an object containing a set of call arguments – is the same).
Every user-defined dispatcher should eventually use dispatcher-delegate in order to identify another dispatcher to pass control along to. In this case, it delegates immediately to boot-value – meaning it really is nothing except a wrapper around the boot-value built-in dispatcher.
The sub identity contains a single static occurrence of the dispatch op. Given we call the sub twice, we will encounter this op twice at runtime, but the two times are very different.
The first time is the “record” phase. The arguments are formed into a capture and the callback runs, which in turn passes it along to the boot-value dispatcher, which produces the result. This results in an extremely simple dispatch program, which says that the result should be the first argument in the capture. Since there’s no guards, this will always be a valid result.
The second time we encounter the dispatch op, it already has a dispatch program recorded there, so we are in run mode. Turning on a debugging mode in the MoarVM source, we can see the dispatch program that results looks like this:
Dispatch program (1 temporaries)
Ops:
Load argument 0 into temporary 0
Set result object value from temporary 0
That is, it reads argument 0 into a temporary location and then sets that as the result of the dispatch. Notice how there is no mention of the fact that we went through an extra layer of dispatch; those have zero cost in the resulting dispatch program.
Argument captures are immutable. Various VM syscalls exist to transform them into new argument captures with some tweak, for example dropping or inserting arguments. Here’s a further example from the test suite:
nqp::dispatch('boot-syscall', 'dispatcher-register', 'drop-first', -> $capture {
my $capture-derived := nqp::dispatch('boot-syscall', 'dispatcher-drop-arg', $capture, 0);
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-value', $capture-derived);
});
ok(nqp::dispatch('drop-first', 'first', 'second') eq 'second',
'dispatcher-drop-arg works');
This drops the first argument before passing the capture on to the boot-value dispatcher – meaning that it will return the second argument. Glance back at the previous dispatch program for the identity function. Can you guess how this one will look?
Well, here it is:
Dispatch program (1 temporaries)
Ops:
Load argument 1 into temporary 0
Set result string value from temporary 0
Again, while in the record phase of such a dispatcher we really do create capture objects and make a dispatcher delegation, the resulting dispatch program is far simpler.
Here’s a slightly more involved example:
my $target := -> $x { $x + 1 }
nqp::dispatch('boot-syscall', 'dispatcher-register', 'call-on-target', -> $capture {
my $capture-derived := nqp::dispatch('boot-syscall',
'dispatcher-insert-arg-literal-obj', $capture, 0, $target);
nqp::dispatch('boot-syscall', 'dispatcher-delegate',
'boot-code-constant', $capture-derived);
});
sub cot() { nqp::dispatch('call-on-target', 49) }
ok(cot() == 50,
'dispatcher-insert-arg-literal-obj works at start of capture');
ok(cot() == 50,
'dispatcher-insert-arg-literal-obj works at start of capture after link too');
Here, we have a closure stored in a variable $target. We insert it as the first argument of the capture, and then delegate to boot-code-constant, which will invoke that code object and pass the other dispatch arguments to it. Once again, at the record phase, we really do something like:
And the resulting dispatch program? It’s this:
Dispatch program (1 temporaries)
Ops:
Load collectable constant at index 0 into temporary 0
Skip first 0 args of incoming capture; callsite from 0
Invoke MVMCode in temporary 0
That is, load the constant bytecode handle that we’re going to invoke, set up the args (which are in this case equal to those of the incoming capture), and then invoke the bytecode with those arguments. The argument shuffling is, once again, gone. In general, whenever the arguments we do an eventual bytecode invocation with are a tail of the initial dispatch arguments, the arguments transform becomes no more than a pointer addition.
All of the dispatch programs seen so far have been unconditional: once recorded at a given callsite, they shall always be used. The big missing piece to make such a mechanism have practical utility is guards. Guards assert properties such as the type of an argument or if the argument is definite (Int:D) or not (Int:U).
Here’s a somewhat longer test case, with some explanations placed throughout it.
# A couple of classes for test purposes
my class C1 { }
my class C2 { }
# A counter used to make sure we're only invokving the dispatch callback as
# many times as we expect.
my $count := 0;
# A type-name dispatcher that maps a type into a constant string value that
# is its name. This isn't terribly useful, but it is a decent small example.
nqp::dispatch('boot-syscall', 'dispatcher-register', 'type-name', -> $capture {
# Bump the counter, just for testing purposes.
$count++;
# Obtain the value of the argument from the capture (using an existing
# MoarVM op, though in the future this may go away in place of a syscall)
# and then obtain the string typename also.
my $arg-val := nqp::captureposarg($capture, 0);
my str $name := $arg-val.HOW.name($arg-val);
# This outcome is only going to be valid for a particular type. We track
# the argument (which gives us an object back that we can use to guard
# it) and then add the type guard.
my $arg := nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $capture, 0);
nqp::dispatch('boot-syscall', 'dispatcher-guard-type', $arg);
# Finally, insert the type name at the start of the capture and then
# delegate to the boot-constant dispatcher.
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-constant',
nqp::dispatch('boot-syscall', 'dispatcher-insert-arg-literal-str',
$capture, 0, $name));
});
# A use of the dispatch for the tests. Put into a sub so there's a single
# static dispatch op, which all dispatch programs will hang off.
sub type-name($obj) {
nqp::dispatch('type-name', $obj)
}
# Check with the first type, making sure the guard matches when it should
# (although this test would pass if the guard were ignored too).
ok(type-name(C1) eq 'C1', 'Dispatcher setting guard works');
ok($count == 1, 'Dispatch callback ran once');
ok(type-name(C1) eq 'C1', 'Can use it another time with the same type');
ok($count == 1, 'Dispatch callback was not run again');
# Test it with a second type, both record and run modes. This ensures the
# guard really is being checked.
ok(type-name(C2) eq 'C2', 'Can handle polymorphic sites when guard fails');
ok($count == 2, 'Dispatch callback ran a second time for new type');
ok(type-name(C2) eq 'C2', 'Second call with new type works');
# Check that we can use it with the original type too, and it has stacked
# the dispatch programs up at the same callsite.
ok(type-name(C1) eq 'C1', 'Call with original type still works');
ok($count == 2, 'Dispatch callback only ran a total of 2 times');
This time two dispatch programs get produced, one for C1:
Dispatch program (1 temporaries)
Ops:
Guard arg 0 (type=C1)
Load collectable constant at index 1 into temporary 0
Set result string value from temporary 0
And another for C2:
Dispatch program (1 temporaries)
Ops:
Guard arg 0 (type=C2)
Load collectable constant at index 1 into temporary 0
Set result string value from temporary 0
Once again, no leftovers from capture manipulation, tracking, or dispatcher delegation; the dispatch program does a type guard against an argument, then produces the result string. The whole call to $arg-val.HOW.name($arg-val) is elided, the dispatcher we wrote encoding the knowledge – in a way that the VM can understand – that a type’s name can be considered immutable.
This example is a bit contrived, but now consider that we instead look up a method and guard on the invocant type: that’s a method cache! Guard the types of more of the arguments, and we have a multi cache! Do both, and we have a multi-method cache.
The latter is interesting in so far as both the method dispatch and the multi dispatch want to guard on the invocant. In fact, in MoarVM today there will be two such type tests until we get to the point where the specializer does its work and eliminates these duplicated guards. However, the new dispatcher does not treat the dispatcher-guard-type as a kind of imperative operation that writes a guard into the resultant dispatch program. Instead, it declares that the argument in question must be guarded. If some other dispatcher already did that, it’s idempotent. The guards are emitted once all dispatch programs we delegate through, on the path to a final outcome, have had their say.
Fun aside: those being especially attentive will have noticed that the dispatch mechanism is used as part of implementing new dispatchers too, and indeed, this ultimately will mean that the specializer can specialize the dispatchers and have them JIT-compiled into something more efficient too. After all, from the perspective of MoarVM, it’s all just bytecode to run; it’s just that some of it is bytecode that tells the VM how to execute Raku programs more efficiently!
A resumable dispatcher needs to do two things:
When a resumption happens, the resume callback will be called, with any arguments for the resumption. It can also obtain the resume initialization state that was set in the dispatch callback. The resume initialization state contains the things needed in order to continue with the dispatch the first time it is resumed. We’ll take a look at how this works for method dispatch to see a concrete example. I’ll also, at this point, switch to looking at the real Rakudo dispatchers, rather than simplified test cases.
The Rakudo dispatchers take advantage of delegation, duplicate guards, and capture manipulations all having no runtime cost in the resulting dispatch program to, in my mind at least, quite nicely factor what is a somewhat involved dispatch process. There are multiple entry points to method dispatch: the normal boring $obj.meth(), the qualified $obj.Type::meth(), and the call me maybe $obj.?meth(). These have common resumption semantics – or at least, they can be made to provided we always carry a starting type in the resume initialization state, which is the type of the object that we do the method dispatch on.
Here is the entry point to dispatch for a normal method dispatch, with the boring details of reporting missing method errors stripped out.
# A standard method call of the form $obj.meth($arg); also used for the
# indirect form $obj."$name"($arg). It receives the decontainerized invocant,
# the method name, and the the args (starting with the invocant including any
# container).
nqp::dispatch('boot-syscall', 'dispatcher-register', 'raku-meth-call', -> $capture {
# Try to resolve the method call using the MOP.
my $obj := nqp::captureposarg($capture, 0);
my str $name := nqp::captureposarg_s($capture, 1);
my $meth := $obj.HOW.find_method($obj, $name);
# Report an error if there is no such method.
unless nqp::isconcrete($meth) {
!!! 'Error reporting logic elided for brevity';
}
# Establish a guard on the invocant type and method name (however the name
# may well be a literal, in which case this is free).
nqp::dispatch('boot-syscall', 'dispatcher-guard-type',
nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $capture, 0));
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal',
nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $capture, 1));
# Add the resolved method and delegate to the resolved method dispatcher.
my $capture-delegate := nqp::dispatch('boot-syscall',
'dispatcher-insert-arg-literal-obj', $capture, 0, $meth);
nqp::dispatch('boot-syscall', 'dispatcher-delegate',
'raku-meth-call-resolved', $capture-delegate);
});
Now for the resolved method dispatcher, which is where the resumption is handled. First, let’s look at the normal dispatch callback (the resumption callback is included but empty; I’ll show it a little later).
# Resolved method call dispatcher. This is used to call a method, once we have
# already resolved it to a callee. Its first arg is the callee, the second and
# third are the type and name (used in deferral), and the rest are the args to
# the method.
nqp::dispatch('boot-syscall', 'dispatcher-register', 'raku-meth-call-resolved',
# Initial dispatch
-> $capture {
# Save dispatch state for resumption. We don't need the method that will
# be called now, so drop it.
my $resume-capture := nqp::dispatch('boot-syscall', 'dispatcher-drop-arg',
$capture, 0);
nqp::dispatch('boot-syscall', 'dispatcher-set-resume-init-args', $resume-capture);
# Drop the dispatch start type and name, and delegate to multi-dispatch or
# just invoke if it's single dispatch.
my $delegate_capture := nqp::dispatch('boot-syscall', 'dispatcher-drop-arg',
nqp::dispatch('boot-syscall', 'dispatcher-drop-arg', $capture, 1), 1);
my $method := nqp::captureposarg($delegate_capture, 0);
if nqp::istype($method, Routine) && $method.is_dispatcher {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-multi', $delegate_capture);
}
else {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-invoke', $delegate_capture);
}
},
# Resumption
-> $capture {
... 'Will be shown later';
});
There’s an arguable cheat in raku-meth-call: it doesn’t actually insert the type object of the invocant in place of the invocant. It turns out that it doesn’t really matter. Otherwise, I think the comments (which are to be found in the real implementation also) tell the story pretty well.
One important point that may not be clear – but follows a repeating theme – is that the setting of the resume initialization state is also more of a declarative rather than an imperative thing: there isn’t a runtime cost at the time of the dispatch, but rather we keep enough information around in order to be able to reconstruct the resume initialization state at the point we need it. (In fact, when we are in the run phase of a resume, we don’t even have to reconstruct it in the sense of creating a capture object.)
Now for the resumption. I’m going to present a heavily stripped down version that only deals with the callsame semantics (the full thing has to deal with such delights as lastcall and nextcallee too). The resume initialization state exists to seed the resumption process. Once we know we actually do have to deal with resumption, we can do things like calculating the full list of methods in the inheritance graph that we want to walk through. Each resumable dispatcher gets a single storage slot on the call stack that it can use for its state. It can initialize this in the first step of resumption, and then update it as we go. Or more precisely, it can set up a dispatch program that will do this when run.
A linked list turns out to be a very convenient data structure for the chain of candidates we will walk through. We can work our way through a linked list by keeping track of the current node, meaning that there need only be a single thing that mutates, which is the current state of the dispatch. The dispatch program mechanism also provides a way to read an attribute from an object, and that is enough to express traversing a linked list into the dispatch program. This also means zero allocations.
So, without further ado, here is the linked list (rather less pretty in NQP, the restricted Raku subset, than it would be in full Raku):
# A linked list is used to model the state of a dispatch that is deferring
# through a set of methods, multi candidates, or wrappers. The Exhausted class
# is used as a sentinel for the end of the chain. The current state of the
# dispatch points into the linked list at the appropriate point; the chain
# itself is immutable, and shared over (runtime) dispatches.
my class DeferralChain {
has $!code;
has $!next;
method new($code, $next) {
my $obj := nqp::create(self);
nqp::bindattr($obj, DeferralChain, '$!code', $code);
nqp::bindattr($obj, DeferralChain, '$!next', $next);
$obj
}
method code() { $!code }
method next() { $!next }
};
my class Exhausted {};
And finally, the resumption handling.
nqp::dispatch('boot-syscall', 'dispatcher-register', 'raku-meth-call-resolved',
# Initial dispatch
-> $capture {
... 'Presented earlier;
},
# Resumption. The resume init capture's first two arguments are the type
# that we initially did a method dispatch against and the method name
# respectively.
-> $capture {
# Work out the next method to call, if any. This depends on if we have
# an existing dispatch state (that is, a method deferral is already in
# progress).
my $init := nqp::dispatch('boot-syscall', 'dispatcher-get-resume-init-args');
my $state := nqp::dispatch('boot-syscall', 'dispatcher-get-resume-state');
my $next_method;
if nqp::isnull($state) {
# No state, so just starting the resumption. Guard on the
# invocant type and name.
my $track_start_type := nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $init, 0);
nqp::dispatch('boot-syscall', 'dispatcher-guard-type', $track_start_type);
my $track_name := nqp::dispatch('boot-syscall', 'dispatcher-track-arg', $init, 1);
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal', $track_name);
# Also guard on there being no dispatch state.
my $track_state := nqp::dispatch('boot-syscall', 'dispatcher-track-resume-state');
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal', $track_state);
# Build up the list of methods to defer through.
my $start_type := nqp::captureposarg($init, 0);
my str $name := nqp::captureposarg_s($init, 1);
my @mro := nqp::can($start_type.HOW, 'mro_unhidden')
?? $start_type.HOW.mro_unhidden($start_type)
!! $start_type.HOW.mro($start_type);
my @methods;
for @mro {
my %mt := nqp::hllize($_.HOW.method_table($_));
if nqp::existskey(%mt, $name) {
@methods.push(%mt{$name});
}
}
# If there's nothing to defer to, we'll evaluate to Nil (just don't set
# the next method, and it happens below).
if nqp::elems(@methods) >= 2 {
# We can defer. Populate next method.
@methods.shift; # Discard the first one, which we initially called
$next_method := @methods.shift; # The immediate next one
# Build chain of further methods and set it as the state.
my $chain := Exhausted;
while @methods {
$chain := DeferralChain.new(@methods.pop, $chain);
}
nqp::dispatch('boot-syscall', 'dispatcher-set-resume-state-literal', $chain);
}
}
elsif !nqp::istype($state, Exhausted) {
# Already working through a chain of method deferrals. Obtain
# the tracking object for the dispatch state, and guard against
# the next code object to run.
my $track_state := nqp::dispatch('boot-syscall', 'dispatcher-track-resume-state');
my $track_method := nqp::dispatch('boot-syscall', 'dispatcher-track-attr',
$track_state, DeferralChain, '$!code');
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal', $track_method);
# Update dispatch state to point to next method.
my $track_next := nqp::dispatch('boot-syscall', 'dispatcher-track-attr',
$track_state, DeferralChain, '$!next');
nqp::dispatch('boot-syscall', 'dispatcher-set-resume-state', $track_next);
# Set next method, which we shall defer to.
$next_method := $state.code;
}
else {
# Dispatch already exhausted; guard on that and fall through to returning
# Nil.
my $track_state := nqp::dispatch('boot-syscall', 'dispatcher-track-resume-state');
nqp::dispatch('boot-syscall', 'dispatcher-guard-literal', $track_state);
}
# If we found a next method...
if nqp::isconcrete($next_method) {
# Call with same (that is, original) arguments. Invoke with those.
# We drop the first two arguments (which are only there for the
# resumption), add the code object to invoke, and then leave it
# to the invoke or multi dispatcher.
my $just_args := nqp::dispatch('boot-syscall', 'dispatcher-drop-arg',
nqp::dispatch('boot-syscall', 'dispatcher-drop-arg', $init, 0),
0);
my $delegate_capture := nqp::dispatch('boot-syscall',
'dispatcher-insert-arg-literal-obj', $just_args, 0, $next_method);
if nqp::istype($next_method, Routine) && $next_method.is_dispatcher {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-multi',
$delegate_capture);
}
else {
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'raku-invoke',
$delegate_capture);
}
}
else {
# No method, so evaluate to Nil (boot-constant disregards all but
# the first argument).
nqp::dispatch('boot-syscall', 'dispatcher-delegate', 'boot-constant',
nqp::dispatch('boot-syscall', 'dispatcher-insert-arg-literal-obj',
$capture, 0, Nil));
}
});
That’s quite a bit to take in, and quite a bit of code. Remember, however, that this is only run for the record phase of a dispatch resumption. It also produces a dispatch program at the callsite of the callsame, with the usual guards and outcome. Implicit guards are created for the dispatcher that we are resuming at that point. In the most common case this will end up monomorphic or bimorphic, although situations involving nestings of multiple dispatch or method dispatch could produce a more morphic callsite.
The design I’ve picked forces resume callbacks to deal with two situations: the first resumption and the latter resumptions. This is not ideal in a couple of ways:
Only the second of these really matters. The reason for the non-uniformity is to make sure that the overwhelming majority of calls, which never lead to a dispatch resumption, incur no per-dispatch cost for a feature that they never end up using. If the result is a little more cost for those using the feature, so be it. In fact, early benchmarking shows callsame with wrap and method calls seems to be up to 10 times faster using the new dispatcher than in current Rakudo, and that’s before the specializer understands enough about it to improve things further!
Everything I’ve discussed above is implemented, except that I may have given the impression somewhere that multiple dispatch is fully implemented using the new dispatcher, and that is not the case yet (no handling of where clauses and no dispatch resumption support).
Getting the missing bits of multiple dispatch fully implemented is the obvious next step. The other missing semantic piece is support for callwith and nextwith, where we wish to change the arguments that are being used when moving to the next candidate. A few other minor bits aside, that in theory will get all of the Raku dispatch semantics at least supported.
Currently, all standard method calls ($obj.meth()) and other calls (foo() and $foo()) go via the existing dispatch mechanism, not the new dispatcher. Those will need to be migrated to use the new dispatcher also, and any bugs that are uncovered will need fixing. That will get things to the point where the new dispatcher is semantically ready.
After that comes performance work: making sure that the specializer is able to deal with dispatch program guards and outcomes. The goal, initially, is to get steady state performance of common calling forms to perform at least as well as in the current master branch of Rakudo. It’s already clear enough there will be some big wins for some things that to date have been glacial, but it should not come at the cost of regression on the most common kinds of dispatch, which have received plenty of optimization effort before now.
Furthermore, NQP – the restricted form of Raku that the Rakudo compiler and other bits of the runtime guts are written in – also needs to be migrated to use the new dispatcher. Only when that is done will it be possible to rip out the current method cache, multiple dispatch cache, and so forth from MoarVM.
An open question is how to deal with backends other than MoarVM. Ideally, the new dispatch mechanism will be ported to those. A decent amount of it should be possible to express in terms of the JVM’s invokedynamic (and this would all probably play quite well with a Truffle-based Raku implementation, although I’m not sure there is a current active effort in that area).
While my current focus is to ship a Rakudo and MoarVM release that uses the new dispatcher mechanism, that won’t be the end of the journey. Some immediate ideas:
handles (delegation) and FALLBACK (handling missing method call) mechanisms can be made to perform better using the new dispatcherassuming – used to curry or otherwise prime arguments for a routine – is not ideal, and an implementation that takes advantage of the argument rewriting capabilities of the new dispatcher would likely perform a great deal betterSome new language features may also be possible to provide in an efficient way with the help of the new dispatch mechanism. For example, there’s currently not a reliable way to try to invoke a piece of code, just run it if the signature binds, or to do something else if it doesn’t. Instead, things like the Cro router have to first do a trial bind of the signature, and then do the invoke, which makes routing rather more costly. There’s also the long suggested idea of providing pattern matching via signatures with the when construct (for example, when * -> ($x) {}; when * -> ($x, *@tail) { }), which is pretty much the same need, just in a less dynamic setting.
Working on the new dispatch mechanism has been a longer journey than I first expected. The resumption part of the design was especially challenging, and there’s still a few important details to attend to there. Something like four potential approaches were discarded along the way (although elements of all of them influenced what I’ve described in this post). Abstractions that hold up are really, really, hard.
I also ended up having to take a couple of months away from doing Raku work at all, felt a bit crushed during some others, and have been juggling this with the equally important RakuAST project (which will be simplified by being able to assume the presence of the new dispatcher, and also offers me a range of softer Raku hacking tasks, whereas the dispatcher work offers few easy pickings).
Given all that, I’m glad to finally be seeing the light at the end of the tunnel. The work that remains is enumerable, and the day we ship a Rakudo and MoarVM release using the new dispatcher feels a small number of months away (and I hope writing that is not tempting fate!)
The new dispatcher is probably the most significant change to MoarVM since I founded it, in so far as it sees us removing a bunch of things that have been there pretty much since the start. RakuAST will also deliver the greatest architectural change to the Rakudo compiler in a decade. Both are an opportunity to fold years of learning things the hard way into the runtime and compiler. I hope when I look back at it all in another decade’s time, I’ll at least feel I made more interesting mistakes this time around.
Many years back, Larry Wall shared his thesis on the nature of scripting. Since recently even Java gained 'script' support I thought it would be fitting to revisit the topic, and hopefully relevant to the perl and raku language community.
The weakness of Larry's treatment (which, to be fair to the author, I think is more intended to be enlightening than to be complete) is the contrast of scripting with programming. This contrast does not permit a clear separation because scripts are programs. That is to say, no matter how long or short, scripts are written commands for a machine to execute, and I think that's a pretty decent definition of a program in general.
A more useful contrast - and, I think, the intended one - is between scripts and other sorts of programs, because that allows us to compare scripting (writing scripts) with 'programming' (writing non-script programs). And to do that we need to know what other sorts of programs there are.
The short version of that answer is - systems and applications, and a bunch of other things that aren't really relevant to the working programmer, like (embedded) control algorithms, spreadsheets and database queries. (The definition I provided above is very broad, by design, because I don't want to get stuck on boundary questions). Most programmers write applications, some write systems, virtually all write scripts once in a while, though plenty of people who aren't professional programmers also write scripts.
I think the defining features of applications and systems are, respectively:
Consider for instance a mail client (like thunderbird) in comparison to a mailer daemon (like sendmail) - one provides an interface to read and write e-mails (the model) and the other provides functionality to send that e-mail to other servers.
Note that under this (again, broad) definition, libraries are also system software, which makes sense, considering that their users are developers (just as for, say, PostgreSQL) who care about things like performance, reliability, and correctness. Incidentally, libraries as well as 'typical' system software (such as database engines and operating system kernels) tend to be written in languages like C and C++ for much the same reasons.
What then, are the differences between scripts, applications, and systems? I think the following is a good list:
Obviously these distinctions aren't really binary - 'short' versus 'long', 'ad-hoc' versus 'general purpose' - and can't be used to conclusively settle the question whether something is a script or an application. (If, indeed, that question ever comes up). More important is that for the 10 or so scripts I've written over the past year - some professionally, some not - all or most of these properties held, and I'd be surprised if the same isn't true for most readers.
And - finally coming at the point that I'm trying to make today - these features point to a specific niche of programs more than to a specific technology (or set of technologies). To be exact, scripts are (mostly) short, custom programs to automate ad-hoc tasks, tasks that are either to specific or too small to develop and distribute another program for.
This has further implications on the preferred features of a scripting language (taken to mean, a language designed to enable the development of scripts). In particular:
This niche doesn't always exist. In computing environments where everything of interest is adequately captured by an application, or which lacks the ability to effectively automate ad-hoc tasks (I'm thinking in particular of Windows before PowerShell), the practice of scripting tends to not develop. Similarily, in a modern 'cloud' environment, where system setup is controlled by a state machine hosted by another organization, scripting doesn't really have much of a future.
To put it another way, scripting only thrives in an environment that has a lot of 'scriptable' tasks; meaning tasks for which there isn't already a pre-made solution available, environments that have powerful facilities available for a script to access, and whose users are empowered to automate those tasks. Such qualities are common on Unix/Linux 'workstations' but rather less so on smartphones and (as noted before) cloud computing environments.
Truth be told I'm a little worried about that development. I could point to, and expound on, the development and popularity of languages like go and rust, which aren't exactly scripting languages, or the replacement of Javascript with TypeScript, to make the point further, but I don't think that's necessary. At the same time I could point to the development of data science as a discipline to demonstrate that scripting is alive and well (and indeed perhaps more economically relevant than before).
What should be the conclusion for perl 5/7 and raku? I'm not quite sure, mostly because I'm not quite sure whether the broader perl/raku community would prefer their sister languages to be scripting or application languages. (As implied above, I think the Python community chose that they wanted Python 3 to be an application language, and this was not without consequences to their users).
Raku adds a number of features common to application languages (I'm thinking of it's powerful type system in particular), continuing a trend that perl 5 arguably pioneered. This is indeed a very powerful strategy - a language can be introduced for scripts and some of those scripts are then extended into applications (or even systems), thereby ensuring its continued usage. But for it to work, a new perl family language must be introduced on its scripting merits, and there must be a plentiful supply of scriptable tasks to automate, some of which - or a combination of which - grow into an application.
For myself, I would like to see scripting have a bright future. Not just because scripting is the most accessible form of programming, but also because an environment that permits, even requires scripting, is one were not all interesting problems have been solved, one where it's users ask it to do tasks so diverse that there isn't an app for that, yet. One where the true potential of the wonderful devices that surround is can be explored.
In such a world there might well be a bright future for scripting.
I’d like to thank everyone who voted for me in the recent Raku Steering Council elections. By this point, I’ve been working on the language for well over a decade, first to help turn a language design I found fascinating into a working implementation, and since the Christmas release to make that implementation more robust and performant. Overall, it’s been as fun as it has been challenging – in a large part because I’ve found myself sharing the journey with a lot of really great people. I’ve also tried to do my bit to keep the community around the language kind and considerate. Receiving a vote from around 90% of those who participated in the Steering Council elections was humbling.
Alas, I’ve today submitted my resignation to the Steering Council, on personal health grounds. For the same reason, I’ll be taking a step back from Raku core development (Raku, MoarVM, language design, etc.) Please don’t worry too much; I’ll almost certainly be fine. It may be I’m ready to continue working on Raku things in a month or two. It may also be longer. Either way, I think Raku will be better off with a fully sized Steering Council in place, and I’ll be better off without the anxiety that I’m holding a role that I’m not in a place to fulfill.
 |
| Both ADD and SUB refer to the same LOAD node |
 |
| The DO node is inserted for the LET operator. It ensures that the value of the LOAD node is computed before the reference in either branch |
I want to revive Carl Mäsak's Coding Contest as a crowd-sourced contest.
The contest will be in four phases:
For the first phase, development of tasks, I am looking for volunteers who come up with coding tasks collaboratively. Sadly, these volunteers, including myself, will be excluded from participating in the second phase.
I am looking for tasks that ...
This is non-trivial, so I'd like to have others to discuss things with, and to come up with some more tasks.
If you want to help with task creation, please send an email to [email protected], stating your intentions to help, and your freenode IRC handle (optional).
There are other ways to help too:
In these cases you can use the same email address to contact me,
or use IRC (moritz on freenode) or twitter.
After a perilous drive up a steep, narrow, winding road from Lake Geneva we arrived at an attractive Alpine village (Villars-sur-Ollon) to meet with fellow Perl Mongers in a small restaurant. There followed much talk and a little clandestine drinking of exotic spirits including Swiss whisky. The following morning walking to the conference venue there was an amazing view of mountain ranges. On arrival I failed to operate the Nespresso machine which I later found was due to it simply being off. Clearly software engineers should never try to use hardware. At least after an evening of drinking.
Wendy’s stall was piled high with swag including new Bailador (Perl 6 dancer like framework) stickers, a Shadowcat booklet about Perl 6 and the new O’Reilly “Thinking in Perl 6″. Unfortunately she had sold out of Moritz’s book “Perl 6 Fundamentals” (although there was a sample display copy present). Thankfully later that morning I discovered I had a £3 credit on Google Play Books so I bought the ebook on my phone.
The conference started early with Damian Conway’s Three Little Words. These were “has”, “class” and “method” from Perl 6 which he liked so much that he had added them to Perl 5 with his “Dios” – “Declarative Inside-Out Syntax” module. PPI wasn’t fast enough so he had to replace it with a 50,000 character regex PPR. Practical everyday modules mentioned included Regexp::Optimizer and Test::Expr. If the video doesn’t appear shortly on youtube a version of his talk dating from a few weeks earlier is available at https://www.youtube.com/watch?v=ob6YHpcXmTg
Jonathan Worthington returned with his Perl 6 talk on “How does deoptimization help us go faster?” giving us insight into why Perl 6 was slow at the Virtual Machine level (specifically MoarVM). Even apparently simple and fast operations like indexing an array were slow due to powerful abstractions, late binding and many levels of Multiple Dispatch. In short the flexibility and power of such an extensible language also led to slowness due to the complexity of code paths. The AST optimizer helped with this at compile time but itself took time and it could be better to do this at a later compile time (like Just In Time). Even with a simple program reading lines from a file it was very hard to determine statically what types were used (even with type annotations) and whether it was worth optimizing (since the file could be very short).
The solution to these dynamic problems was also dynamic but to see what was happening needed cheap logging of execution which was passed to another thread. This logging is made visible by setting the environment variable MVM_SPESH_LOG to a filename. Better tooling for this log would be a good project for someone.
For execution planning we look for hot (frequently called) code, long blocks of bytecode (slow to run) and consider how many types are used (avoiding “megamorphic” cases with many types which needs many versions of code). Also analysis of the code flow between different code blocks and SSA. Mixins made the optimization particularly problematic.
MoarVM’s Spesh did statistical analysis of the code in order to rewrite it in faster, simpler ways. Guards (cheap check for things like types) were placed to catch cases where it got it wrong and if these were triggered (infrequently) it would deoptimize as well, hence the counterintuitive title since “Deoptimization enables speculation” The slides are at http://jnthn.net/papers/2017-spw-deopt.pdf with the video at https://www.youtube.com/watch?v=3umNn1KnlCY The older and more dull witted of us (including myself) might find the latter part of the video more comprehensible at 0.75 Youtube speed.
After a superb multi-course lunch (the food was probably the best I’d had at any Perl event) we returned promptly to hear Damian talk of “Everyday Perl 6”. He pointed out that it wasn’t necessary to code golf obfuscated extremes of Perl 6 and that the average Perl 5 programmer would see many things simpler in Perl 6. Also a rewrite from 5 to 6 might see something like 25% fewer lines of code since 6 was more expressive in syntax (as well as more consistent) although performance problems remained (and solutions in progress as the previous talk had reminded us).
Next Liz talked of a “gross” (in the numerical sense of 12 x 12 rather than the American teen sense) of Perl 6 Weeklies as she took us down memory lane to 2014 (just about when MoarVM was launched and when unicode support was poor!) with some selected highlights and memories of Perl 6 developers of the past (and hopefully future again!). Her talk was recorded at https://www.youtube.com/watch?v=418QCTXmvDU
Cal then spoke of Perl 6 maths which he thought was good with its Rats and FatRats but not quite good enough and his ideas of fixing it. On the following day he showed us he had started some TDD work on TrimRats. He also told us that Newton’s Method wasn’t very good but generated a pretty fractal. See https://www.youtube.com/watch?v=3na_Cx-anvw
Lee spoke about how to detect Perl 5 memory leaks with various CPAN modules and his examples are at https://github.com/leejo/Perl_memory_talk
The day finished with Lightning Talks and a barbecue at givengain — a main sponsor.
On the second day I noticed the robotic St Bernards dog in a tourist shop window had come to life.
Damian kicked off the talks with my favourite of his talks, “Standing on the Shoulders of Giants”, starting with the Countess of Lovelace and her Bernoulli number program. This generated a strange sequence with many zeros. The Perl 6 version since it used rational numbers not floating point got the zeros right whereas the Perl 5 version initially suffered from floating point rounding errors (which are fixable).
Among other things he showed us how to define a new infix operator in Perl 6. He also showed us a Perl 6 sort program that looked exactly like LISP even down to the Lots of Irritating Superfluous Parentheses. I think this was quicksort (he certainly showed us a picture of Sir Tony Hoare at some point). Also a very functional (Haskell-like) equivalent with heavy use of P6 Multiple Dispatch. Also included was demonstration of P6 “before” as a sort of typeless/multi-type comparison infix. Damian then returned to his old favourite of Quantum Computing.
My mind and notes got a bit jumbled at this point but I particularly liked the slide that explained how factorisation could work by observing the product of possible inputs since this led to a collapse that revealed the factors. To do this on RSA etc., of course, needs real hardware support which probably only the NSA and friends have (?). Damian’s code examples are at http://www.bit.do/Perl6SOG with an earlier version of his talk at https://www.youtube.com/watch?v=Nq2HkAYbG5o Around this point there was a road race of classic cars going on outside up the main road into the village and there were car noises in the background that strangely were more relaxing than annoying.
After Quantum Chaos Paul Johnson brought us all back down to ground with an excellent practical talk on modernising legacy Perl 5 applications based on his war stories. Hell, of course, is “Other People’s Code”, often dating from Perl’s early days and lacking documentation and sound engineering.
Often the original developers had long since departed or, in the worse cases, were still there. Adding tests and logging (with stack traces) were particularly useful. As was moving to git (although its steep learning curve meant mentoring was needed) and handling CPAN module versioning with pinto. Many talks had spoken of the Perl 6 future whereas this spoke of the Perl 5 past and present and the work many of us suffer to pay the bills. It’s at https://www.youtube.com/watch?v=4G5EaUNOhR0
Jonathan then spoke of reactive distributed software. A distributed system is an async one where “Is it working?” means “some of it is working but we don’t know which bits”. Good OO design is “tell don’t ask” — you tell remote service to do something for you and not parse the response and do it yourself thus breaking encapsulation. This is particularly important in building well designed distributed systems since otherwise the systems are less responsive and reliable. Reactive (async) works better for distributed software than interactive (blocking or sync).
We saw a table that used a Perl 6 promise for one value and a supply for many values for reactive (async) code and the equivalent (one value) and a Perl 6 Seq for interactive code. A Supply could be used for pub/sub and the Observer Pattern. A Supply could either be live (like broadcast TV) or, for most Perl 6 supplies, on-demand (like Netflix). Then samples of networking (socket) based code were discussed including a web client, web server and SSH::LibSSH (async client bindings often very useful in practical applications like port forwarding)
https://github.com/jnthn/p6-ssh-libssh
Much of the socket code had a pattern of “react { whenever {” blocks with “whenever” as a sort of async loop.He then moved on from sockets to services (using a Supply pipeline) and amazed us by announcing the release of “cro”, a microservices library that even supports HTTP/2 and Websockets, at http://mi.cro.services/. This is installable using Perl 6 by “zef install –/test cro”.
Slides at http://jnthn.net/papers/2017-spw-sockets-services.pdf and video at https://www.youtube.com/watch?v=6CsBDnTUJ3A
Next Lee showed Burp Scanner which is payware but probably the best web vulnerabilities scanner. I wondered if anyone had dare run it on ACT or the hotel’s captive portal.
Wendy did some cheerleading in her “Changing Image of Perl”. An earlier version is at https://www.youtube.com/watch?v=Jl6iJIH7HdA
Sue’s talk was “Spiders, Gophers, Butterflies” although the latter were mostly noticeably absent. She promises me that a successor version of the talk will use them more extensively. Certainly any Perl 6 web spidering code is likely to fit better on one slide than the Go equivalent.
During the lightning talks Timo showed us a very pretty Perl 6 program using his SDL2::Raw to draw an animated square spiral with hypnotic colour cycling type patterns. Also there was a talk by the author about https://bifax.org/bif/— a distributed bug tracking system (which worked offline like git).
Later in the final evening many of us ate and chatted in another restaurant where we witnessed a dog fight being narrowly averted and learnt that Wendy didn’t like Perl 5’s bless for both technical and philosophical reasons.
Time for some old man's reminiscence. Or so it feels when I realize that I've spent more than 10 years involved with the Perl 6 community.
It was February 2007.
I was bored. I had lots of free time (crazy to imagine that now...), and I spent some of that answering (Perl 5) questions on perlmonks. There was a category of questions where I routinely had no good answers, and those were related to threads. So I decided to play with threads, and got frustrated pretty quickly.
And then I remember that a friend in school had told me (about four years earlier) that there was this Perl 6 project that wanted to do concurrency really well, and even automatically parallelize some stuff. And this was some time ago, maybe they had gotten anywhere?
So I searched the Internet, and found out about Pugs, a Perl 6 compiler written in Haskell. And I wanted to learn more, but some of the links to the presentations were dead. I joined the #perl6 IRC channel to report the broken link.
And within three minutes I got a "thank you" for the report, the broken links were gone, and I had an invitation for a commit bit to the underlying SVN repo.
I stayed.
Those were they wild young days of Perl 6 and Pugs. Audrey Tang was pushing Pugs (and Haskell) very hard, and often implemented a feature within 20 minutes after somebody mentioned it. Things were unstable, broken often, and usually fixed quickly. No idea was too crazy to be considered or even implemented.
We had bots that evaluated Perl 6 and Haskell code, and gave the result directly on IRC. There were lots of cool (and sometimes somewhat frightening) automations, for example for inviting others to the SVN repo, to the shared hosting system (called feather), for searching SVN logs and so on. Since git was still an obscure and very unusable, people tried to use SVK, an attempt to implement a decentralized version control system on top of of the SVN protocol.
Despite some half-hearted attempts, I didn't really make inroads into compiler developments. Having worked with neither Haskell nor compilers before proved to be a pretty steep step. Instead I focused on some early modules, documentation, tests, and asking and answering questions. When the IRC logger went offline for a while, I wrote my own, which is still in use today.
I felt at home in that IRC channel and the community. When the community asked for mentors for the Google Summer of Code project, I stepped up. The project was a revamp of the Perl 6 test suite, and to prepare for mentoring task, I decided to dive deeper. That made me the maintainer of the test suite.
I can't recount a full history of Perl 6 projects during that time range, but I want to reflect on some projects that I considered my pet projects, at least for some time.
It is not quite clear from this (very selected) timeline, but my Perl 6 related activity dropped around 2009 or 2010. This is when I started to work full time, moved in with my girlfriend (now wife), and started to plan a family.
The technologies and ideas in Perl 6 are fascinating, but that's not what kept me. I came for the technology, but stayed for the community.
There were and are many great people in the Perl 6 community, some of whom I am happy to call my friends. Whenever I get the chance to attend a Perl conference, workshop or hackathon, I find a group of Perl 6 hackers to hang out and discuss with, and generally have a good time.
Four events stand out in my memory. In 2010 I was invited to the Open Source Days in Copenhagen. I missed most of the conference, but spent a day or two with (if memory serve right) Carl Mäsak, Patrick Michaud, Jonathan Worthington and Arne Skjærholt. We spent some fun time trying to wrap our minds around macros, the intricacies of human and computer language, and Japanese food. (Ok, the last one was easy). Later the same year, I attended my first YAPC::EU in Pisa, and met most of the same crowd again -- this time joined by Larry Wall, and over three or four days. I still fondly remember the Perl 6 hallway track from that conference. And 2012 I flew to Oslo for a Perl 6 hackathon, with a close-knit, fabulous group of Perl 6 hackers. Finally, the Perl Reunification Summit in the beautiful town of Perl in Germany, which brought together Perl 5 and Perl 6 hackers in a very relaxed atmosphere.
For three of these four events, different private sponsors from the Perl and Perl 6 community covered travel and/or hotel costs, with their only motivation being meeting folks they liked, and seeing the community and technology flourish.
The Perl 6 community has evolved a lot over the last ten years, but it is still a very friendly and welcoming place. There are lots of "new" folks (where "new" is everybody who joined after me, of course :D), and a surprising number of the old guard still hang around, some more involved, some less, all of them still very friendly and supportive
I anticipate that my family and other projects will continue to occupy much of my time, and it is unlikely that I'll be writing another Perl 6 book (after the one about regexes) any time soon. But the Perl 6 community has become a second home for me, and I don't want to miss it.
In the future, I see myself supporting the Perl 6 community through infrastructure (community servers, IRC logs, running IRC bots etc.), answering questions, writing a blog article here and there, but mostly empowering the "new" guard to do whatever they deem best.
After about nine months of work, my book Perl 6 Fundamentals is now available for purchase on apress.com and springer.com.
The ebook can be purchased right now, and comes in the epub and PDF formats (with watermarks, but DRM free). The print form can be pre-ordered from Amazon, and will become ready for shipping in about a week or two.
I will make a copy of the ebook available for free for everybody who purchased an earlier version, "Perl 6 by Example", from LeanPub.
The book is aimed at people familiar with the basics of programming; prior
Perl 5 or Perl 6 knowledge is not required. It features a practical example in most chapters (no mammal hierarchies or class Rectangle inheriting from class Shape), ranging from simple input/output and text formatting to plotting with python's matplotlib libraries. Other examples include date and time conversion, a Unicode search tool and a directory size visualization.
I use these examples to explain subset of Perl 6, with many pointers to more
documentation where relevant. Perl 6 topics include the basic lexicographic
structure, testing, input and output, multi dispatch, object orientation, regexes and grammars, usage of modules, functional programming and interaction
with python libraries through Inline::Python.
Let me finish with Larry Wall's description of this book, quoted from his foreword:
It's not just a reference, since you can always find such materials online. Nor is it just a cookbook. I like to think of it as an extended invitation, from a well-liked and well-informed member of our circle, to people like you who might want to join in on the fun. Because joy is what's fundamental to Perl. The essence of Perl is an invitation to love, and to be loved by, the Perl community. It's an invitation to be a participant of the gift economy, on both the receiving and the giving end.
The Perl 6 naming debate has started again. And I guess with good reason. Teaching people that Perl 6 is a Perl, but not the Perl requires too much effort. Two years ago, I didn't believe. Now you're reading a tired man's words.
I'm glad that this time, we're not discussing giving up the "Perl" brand, which still has very positive connotations in my mind, and in many other minds as well.
And yet, I can't bring myself to like "Rakudo Perl 6" as a name. There are two vary shallow reasons for that: Going from two syllables, "Perl six", to five of them, seems a step in the wrong direction. And two, I remember the days when the name was pretty young, and people would misspell it all the time. That seems to have abated, though I don't know why.
But there's also a deeper reason, probably sentimental old man's reason. I remember the days when Pugs was actively developed, and formed the center of a vibrant community. When kp6 and SMOP and all those weird projects were around. And then, just when it looked like there was only a single compiler was around, Stefan O'Rear conjured up niecza, almost single-handedly, and out of thin air. Within months, it was a viable Perl 6 compiler, that people on #perl6 readily recommended.
All of this was born out of the vision that Perl 6 was a language with no single, preferred compiler. Changing the language name to include the compiler name means abandoning this vision. How can we claim to welcome alternative implementations when the commitment to one compiler is right in the language name?
However I can't weigh this loss of vision against a potential gain in popularity. I can't decide if it's my long-term commitment to the name "Perl 6" that makes me resent the new name, or valid objections. The lack of vision mirrors my own state of mind pretty well.
I don't know where this leaves us. I guess I must apologize for wasting your time by publishing this incoherent mess.
Perl 6 is innovative in many ways, and sometimes we don't fully appreciate all the implications, for good or for bad.
There's one I stumbled upon recently: The use of fancy Unicode symbols for built-in stuff. In this case: the `.gist` output of Match objects. For example
my token word { \w+ } say 'abc=def' ~~ /<word> '=' <word>/;produces this output:
｢abc=def｣ word => ｢abc｣ word => ｢def｣
And that's where the problems start. In my current quest to write a book on Perl 6 regexes, I noticed that the PDF that LeanPub generates from my Markdown sources don't correctly display those pesky ｢｣ characters, which are
$ uni -c ｢｣ ｢ - U+0FF62 - HALFWIDTH LEFT CORNER BRACKET ｣ - U+0FF63 - HALFWIDTH RIGHT CORNER BRACKET
When I copied the text from the PDF and pasted into my editor, they showed up correctly, which indicates that the characters are likely missing from the monospace font.
The toolchain allows control over the font used for displaying code, so I tried all the monospace fonts that were available. I tried them in alphabetical order. Among the earlier fonts I tried was Deja Vu Sans Mono, which I use in my terminal, and which hasn't let me down yet. No dice. I arrived at Noto, a font designed to cover all Unicode codepoints. And it didn't work either. So it turns out these two characters are part of some Noto Sans variants, but not of the monospace font.
My terminal, and even some font viewers, use some kind of fallback where they use glyphs from other fonts to render missing characters. The book generation toolchain does not.
The Google Group for Leanpub was somewhat helpful: if I could recommend an Open Source mono space font that fit my needs, they'd likely include it in their toolchain.
So I searched and searched, learning more about fonts than I wanted to know. My circle of geek friends came up with several suggestions, one of them being Iosevka, which actually contains those characters. So now I wait for others to step up, either for LeanPub to include that font, or for the Noto maintainers to create a monospace variant of those characters (and then LeanPub updating their version of the font).
And all of that because Perl 6 was being innovative, and used two otherwise little-used characters as delimiters, in an attempt to avoid collisions between delimiters and content.
(In the mean time I've replaced the two offending characters with ones that look similar. It means the example output is technically incorrect, but at least it's readable).
At YAPC::EU 2010 in Pisa I received a business card with "Rakudo Star" and the
date July 29, 2010 which was the date of the first release -- a week earlier
with a countdown to 1200 UTC. I still have mine, although it has a tea stain
on it and I refreshed my memory over the holidays by listening again to Patrick
Michaud speaking about the launch of Rakudo Star (R*):
https://www.youtube.com/watch?v=MVb6m345J-Q
R* was originally intended as first of a number of distribution releases (as
opposed to a compiler release) -- useable for early adopters but not initially production
Quality. Other names had been considered at the time like Rakudo Beta (rejected as
sounding like "don't use this"!) and amusingly Rakudo Adventure Edition.
Finally it became Rakudo Whatever and Rakudo Star (since * means "whatever"!).
Well over 6 years later and we never did come up with a better name although there
was at least one IRC conversation about it and perhaps "Rakudo Star" is too
well established as a brand at this point anyway. R* is the Rakudo compiler, the main docs, a module installer, some modules and some further docs.
However, one radical change is happening soon and that is a move from panda to
zef as the module installer. Panda has served us well for many years but zef is
both more featureful and more actively maintained. Zef can also install Perl
6 modules off CPAN although the CPAN-side support is in its early days. There
is a zef branch (pull requests welcome!) and a tarball at:
http://pl6anet.org/drop/rakudo-star-2016.12.zef-beta2.tar.gz
Panda has been patched to warn that it will be removed and to advise the use of
zef. Of course anyone who really wants to use panda can reinstall it using zef
anyway.
The modules inside R* haven't changed much in a while. I am considering adding
DateTime::Format (shown by ecosystem stats to be widely used) and
HTTP::UserAgent (probably the best pure perl6 web client library right now).
Maybe some modules should also be removed (although this tends to be more
controversial!). I am also wondering about OpenSSL support (if the library is
available).
p6doc needs some more love as a command line utility since most of the focus
has been on the website docs and in fact some of these changes have impacted
adversely on command line use, eg. under Windows cmd.exe "perl 6" is no longer
correctly displayed by p6doc. I wonder if the website generation code should be
decoupled from the pure docs and p6doc command line (since R* has to ship any
new modules used by the website). p6doc also needs a better and faster search
(using sqlite?). R* also ships some tutorial docs including a PDF generated from perl6intro.com.
We only ship the English one and localisation to other languages could be
useful.
Currently R* is released roughly every three months (unless significant
breakage leads to a bug fix release). Problems tend to happen with the
less widely used systems (Windows and the various BSDs) and also with the
module installers and some modules. R* is useful in spotting these issues
missed by roast. Rakudo itself is still in rapid development. At some point a less frequently
updated distribution (Star LTS or MTS?) will be needed for Linux distribution
packagers and those using R* in production). There are also some question
marks over support for different language versions (6.c and 6.d).
Above all what R* (and Rakudo Perl 6 in general) needs is more people spending
more time working on it! JDFI! Hopefully this blog post might
encourage more people to get involved with github pull requests.
https://github.com/rakudo/star
Feedback, too, in the comments below is actively encouraged.
There is a Release Candidate for Rakudo Star 2016.11 (currently RC2) available at
http://pl6anet.org/drop/
This includes binary installers for Windows and Mac.
Usually Star is released about every three months but last month's release didn't include a Windows installer so there is another release.
I'm hoping to release the final version next weekend and would be grateful if people could try this out on as many systems as possible.
Any feedback email steve *dot* mynott *at* gmail *dot* com
Full draft announce at
https://github.com/rakudo/star/blob/master/docs/announce/2016.11.md
We turned up in Cluj via Wizz Air to probably one of the best pre YAPC parties ever located on three levels on the rooftop of Evozon’s plush city centre offices. We were well supplied with excellent wine, snacks and the local Ursus beer and had many interesting conversations with old friends.
On the first day Tux spoke about his Text::CSV modules for both Perl 5 and 6 on the first day and I did a short talk later in the day on benchmarking Perl 6. Only Nicholas understood my trainspotter joke slide with the APT and Deltic! Sadly my talk clashed with Lee J talking about Git which I wanted to see so I await the youtube version! Jeff G then spoke about Perl 6 and parsing languages such as JavaScript. Sadly I missed Leon T’s Perl 6 talk which I also plan on watching on youtube. Tina M gave an excellent talk on writing command line tools. She also started the lightning talks with an evangelical talk about how tmux was better than screen. Geoffrey A spoke about configuring sudo to run restricted commands in one directory which seemed a useful technique to me. Dave C continued his conference tradition of dusting off his Perl Vogue cover and showing it again. The age of the image was emphasised by the amazingly young looking mst on it. And Stefan S ended with a call for Perl unification.
The main social event was in the courtyard of the main museum off the central square with free food and beer all evening and an impressive light show on the slightly crumbling facade. There were some strange chairs which resembled cardboard origami but proved more comfortable than they looked when I was finally able to sit in one. The quality of the music improved as the evening progressed (or maybe the beer helped) I was amazed to see Perl Mongers actually dancing apparently inspired by the younger Cluj.pm members.
Day Two started with Sawyer’s State of the Velociraptor which he had, sensibly, subcontracted to various leading lights of the Perl Monger community. Sue S (former London.pm leader) was up first with a short and sweet description of London.pm. Todd R talked about Houston.pm. Aaron Crane spoke about the new improved friendlier p5p. Tina about Berlin.pm and the German Perl community site she had written back in the day. This new format worked very well and it was obvious Perl Mongers groups could learn much from each other. Max M followed with a talk about using Perl and ElasticSearch to index websites and documents and Job about accessibility.
1505 had, from the perspective of London.pm, one of the most unfortunate scheduling clashes at YAPC::EU ever, with three titans of London.pm (all former leaders) battling for audience share. I should perhaps tread carefully here lest bias become apparent but the heavyweight Sue Spence was, perhaps treacherously, talking about Go in the big room and Dave Cross and Tom talking about Perl errors and HTML forms respectively in the other rooms. This momentous event should be reproducible by playing all three talks together in separate windows once they are available.
Domm did a great talk on Postgres which made me keen to use this technology again. André W described how he got Perl 6 running on his Sailfish module phone while Larry did a good impression of a microphone stand. I missed most of Lance Wick’s talk but the bit I caught at the end made me eager to watch the whole thing.
Guinevere Nell gave a fascinating lightning talk about agent based economic modelling. Lauren Rosenfield spoke of porting (with permission) a “Python for CS” book to perl 6. Lukas Mai described his journey from Perl to Rust. Lee J talked about photography before Sue encouraged people to break the London.pm website. Outside the talk rooms on their stall Liz and Wendy had some highly cool stuffed toy Camelia butterflies produced by the Beverly Hills Teddy Bear Company and some strange “Camel Balls” bubblegum. At the end of the day Sue cat herded many Mongers to eat at the Enigma Steampunk Bar in central Cluj with the cunning ploy of free beer money (recycled from the previous year’s Sherry money).
The third day started with Larry’s Keynote in which photographs of an incredible American house “Fallingwater” and Chinese characters (including “arse rice”) featured heavily. Sweth C gave a fast and very useful introduction to swift. Nicholas C then confused a room of people for an hour with a mixture of real Perl 5 and 6 and an alternative timeline compete with T shirts. The positive conclusion was that even if the past had been different the present isn’t likely to have been much better for the Perl language family than it is now! Tom spoke about Code Review and Sawyer about new features in Perl 5.24. Later I heard Ilya talk about running Perl on his Raspberry PI Model B and increasing the speed of his application very significantly to compensate for its low speed! And we finished with lightning talks where we heard about the bug tracker OTRS (which was new to me), Job spoke about assistive tech and Nine asked us to ask our bosses for money for Perl development amongst several other talks. We clapped a lot in thanks, since this was clearly a particularly well organised YAPC::EU (due to Amalia and her team!) and left to eat pizza and fly away the next day. Some stayed to visit a salt mine (which looked most impressive from the pictures!) and some stayed longer due to Lufthansa cancelling their flights back!
The meeting first night was in a large beer bar in the centre of Nuremberg.
We went back to the Best Western to find a certain exPumpkin already resident in the bar.
Despite several of the well named Bitburgers we managed to arrive at the
conference venue on time the following morning. Since my knowledge of German was
limited to a C grade 'O' Level last century my review talks will be mostly
limited to English talks. Apologies in advance to those giving German talks
(not unreasonable considering the country). Hopefully other blog posts will
cover these.
Masak spoke about the dialectic between planning (like physics) and chaos (like
biology) in software development.
http://masak.org/carl/gpw-2016-domain-modeling/talk.pdf
Tobias gave a good beginners guide to Perl 6 in German and I was able to follow
most of the slides since I knew more Perl 6 than German and even learnt a thing
or two.
After lunch Stefan told us he was dancing around drunk and naked on the turn of
the 2000s and also about communication between Perl 6 and Perl 5 and back again
via his modules Inline::Perl5 (from Perl 6) -- the most important take away
being that "use Foo::Bar:from<Perl5>" can be used from Perl 6 and "use
Inline::Perl6" from Perl 5. The modules built bridges like those built in the
old school computer game "Lemmings".
http://niner.name/talks/Perl%205%20and%20Perl%206%20-%20a%20great%20team/Perl%205%20and%20Perl%206%20-%20a%20great%20team.odp
Max told us (in German) about his Dancer::SearchApp search
engine which has based on Elastic Search but I was able to follow along on the
English version of his slides on the web.
http://corion.net/talks/dancer-searchapp/dancer-searchapp.en.html
Sue got excited about this. Tina showed us some slides in Vim and her module
to add command line tab completion to script arguments using zsh and bash. I
wondered whether some of her code could be repurposed to add fish shell man
page parsing autocompletion to zsh. She also had a good lightening talk about
Ingy's command line utility for github.
https://github.com/perlpunk/myslides/tree/master/app-spec
Second day started early with Moritz talking about Continuous Delivery which
could mean just delivering to a staging server. He was writing a book about it
at deploybook.com with slides at:
https://deploybook.com/talks/gpw2016-continuous-delivery.pdf
Salve wanted us to write elegant code as a reply to the Perl Jam guy at CCC in
a self confessed "rant".
Sawyer described writing Ref::Util to optimise things like "ref $foo" in a
Hardcore Perl 5 XS/Core talk and Masak told us about his little 007 language
written in Perl 6 as a proof of concept playroom for future Perl 6 extended
macro support and demonstrated code written over lunch in support of this.
http://masak.org/carl/gpw-2016-big-hairy-yaks/talk.pdf
Stefan gave a great talk about CURLI and explained the complexity of what was
intended.
I gave my talk on "Simple Perl 6 Fractals and Concurrency" on Friday. It
started badly with AV issues my side but seemed well received. It was useful
speaking with people about it and I managed to speed things up *after* the talk
and I should have new material for a 2.0 version.
There were very good talks on extracting data from PDFs and writing JSON apis.
https://github.com/mickeyn/PONAPI
looked very interesting and would have saved me much coding at a recent job.
There were some great lightening talks at th end of the day. Sawyer wanted
people to have English slides and gave his talk in Hebrew to stress this.
Things ended Friday night with great food and beer in a local bar.
To me It seemed a particularly good FOSDEM for both for Perl5/6 and
other talks although very crowded as usual and I didn't see the usual
*BSD or Tor stalls. I was stuck by the statistic that there were
about 500 speakers from many thousands of people so of the order of
one speaker per tens of attendees which is very high.
Videos are already starting to appear at
On Saturday I started with Poettering and systemd which was a keynote
and perhaps a little disappointing since he usually is a better
speaker and the audio was a little indistinct. systemd had won being
used by all distros except gentoo and slackware. They were now working
on a dns resolver component which supported DNSSEC although in
practice validating signed zone files would slow down browsing and
currently only 2% of websites had it activated. He didn't mention
strong criticisms of its security by crypto experts such as DJB.
The most amusing talk was Stark's retro running of Postgres on
NetBSD/VAX which exposed some obscure OS bugs and was livened up by a
man in an impressive Postgres Elephant costume appearing. We later
spoke to Mr Elephant who said he was both blind and very hot at the
time. I then went to the Microkernel room to hear about GNU/Hurd
progress from Thibault since this room is usually "OPEN" and he's an
excellent speaker. I noticed even this obscure room was quite crowded
as compared with previous years so I'd guess total attendees this year
were high. He stressed the advantages of running device drivers in
userspace as allowing more user "freedom" to mount fs etc. without
root and improving kernel stability since the drivers could crash and
restart without bringing down the kernel. In previous years he had
talked of his DDE patches allowing linux 2.6 hardware drivers on Hurd
and this year he was using the NetBSD Rump kernel under Hurd to add
sound support with USB support promised. His demo was RMS singing his
song on his Hurd laptop. The irony was he needed to use BSD code on a
GNU/BSD/Hurd system to do it! There had been some work on X86-64 Hurd
but it wasn't there yet since he needed more help from the community.
I then saw some lightening talks (actually 20 mins long) including a
good one on C refactoring.
The Perl dinner on Saturday night featured the usual good food and
conversation and the devroom was on Sunday. Ovid spoke about Perl 6
and its advantages (such as being able to perform maths on floats
correctly). I had a python guy sitting next to me who admitted he had
never been to a Perl talk before so that was a success in reaching
someone new. Will Braswell spoke next about his "Rperl" compiler
which translated his own quite restricted subset (no regexps yet and
no $_) of Perl 5 line by line into C++ in order to run some of the
language shootups benchmarks (a graphical animation of planetary
motion) at increased speed. I'd not seen Will before and he was an
excellent speaker who left me more impressed than I'd expected and I
hope he gets to YAPC::EU in the summer. I saw some non-Perl stuff
next for variety including a good one on the Go debugger Delve which
was aware of the go concurrency and could be used as a basic REPL. I
returned to Perl to see Bart explain some surprisingly simple X86-64
assembly language to do addition and ROT13 which he interfaced with
Perl 6 using NativeCall (although it stuck me that the
CPAN P5NCI module on Perl 5 would have also worked).
Again an excellent talk and a good start to the a
run of some of the best Perl talks I'd ever seen. Stevan Little's talk
was one of the his most amusing ever and perl wasn't really dead.
Sawyer also did an excellent promotion of Perl 5 targeted at the
people who maybe hadn't used it since the early 2000s explaining what
had changed. Liz finished with her autobiographical account of Perl
development and some nice short Perl 6 examples. We all ate again in
the evening together my only regrets being I'd missed the odd talk or
two (which I should be able to watch on video).
MetaCPAN, like the rest of "CPAN", was built assuming the sole context of Perl5. Which is cool until we want to use it for Perl6 and avoid the troubles associated with different namespaces, dist mgmt, etc... To largely avoid and more easily handle these issues for MetaCPAN it's been suggested that we have separate instances. The existing Perl5 instance only needs to be changed to ignore Perl6 distributions. There has already been some breakage because it didn't ignore a Perl6 dist of mine which exists in the Perl5 world:( And the new Perl6 instance will do just the opposite and only look at Perl6 distributions.
In contrast, and relatedly, on CPAN we've designated a special spot for Perl6 distributions in order to keep them separate from the Perl5 dists. This reserved place is a Perl6 subdir in an author's dir (/author/id/*/*/*/Perl6/). Any dists in or under that spot on the fs will be considered a Perl6 dist; valid or invalid. So this is where the Perl6 MetaCPAN will look and the Perl5 instance will not.
Current development is being done on these temporary branches:
And the main dev instance is running on hack.p6c.org. The web end is at http://hack.p6c.org:5001 and the api is at http://hack.p6c.org:5000.
So far the idea has been to iterate on the aforementioned branches and instance until we have something that works sufficiently well. At that point we'll tidy up the branches and submit them for merging. Shortly after that time the hope is that we'll be able to stand up the official Perl6 instance.
The list of requirements for being adequately cooked is:
All of these have been hacked in and are at various degrees of completeness. Next up is testing and fixing bugs until nothing major is left. To that end I've recently loaded up the dev instance with all the distributions from modules.perl6.org. The dist files were generated, very hackily, with https://github.com/jdv/cpan-api/blob/master/test_p6_eco_to_p6_cpan.pl. I also just loaded them all under one user, mine, for simplicity. That load looks like it has problems of its own as well as revealing a bunch of issues. So in the coming days I hope to get that all sorted out.
In the Perl5 world, just in case anyone is unaware, CPAN is a major factor. Its basically the hub of the Perl5 world.
What I am referring to here as CPAN is not just the mirrored collection of 32K+ distributions. Its the ecosystem that's built up around that collection. This ecosystem has many parts, some more important than others depending on who you talk to, but the most important parts to me are:
These are the 5 aspects of "CPAN" that I'd like to see happen for Perl6. One way to get that would be to write the whole thing from scratch in Perl6. While it may sound cool in some sort of dogfoody and/or bootstrappy kind of way to some, it sounds like a lot of work to me and we're a bit strapped for developer resources. Another way would be to add support for Perl6 to the existing CPAN bits. The hope there being, primarily, that it'd be a lot less work. The latter approach is what I've been working on lately. And if we want to refactor ourselves off the Perl5 bits in the future we can take our time doing it; later.
At this time we have:So we can publish Perl6 distributions to CPAN and search that collection. Well, sort of on that last bit. The metacpan prototype instance is not currently tracking CPAN. Its actually been loaded up with Perl6 distributions from the Perl6 module ecosystem (modules.perl6.org) for testing. But hopefully soon we'll have an official Perl6 metacpan instance, separate from the Perl5 instance, that will track CPAN's Perl6 content as it should.
What we need next is:If anyone is interested in working on any of this stuff please stop by #perl6 on freenode. If nobody else is able to help you I'll (jdv79) do my best.
Thanks to those on Freenode IRC/perl6 for help.
Further corrections and expansions welcome either on iRC via pull request to https://github.com/stmuk/glr-html
| pre GLR | GLR |
|
| LIST IS NOW PARCEL |
> say (1,2,3).WHAT (Parcel) |
> say (1,2,3).WHAT (List) |
| LACK OF IMPLICIT LIST FLATTENING |
> my @array = 1,(2,3),4 1 2 3 4 > @array.elems 4 |
my @array = 1,(2,3),4 [1 (2 3) 4] > @array.elems 3 to flatten> my @list := 1, [2, 3], 4 (1 [2 3] 4) > dd @list.flat.list (1, 2, 3, 4) or> my @array = (1,(2,3),4).flat [1 2 3 4] or more complex structures (jnthn++)say gather [[[[["a", "b"], "c"], "a"], "d"], "e"].deepmap(*.take) |
| .lol METHOD REMOVED |
> dd (1,2,3).lol (1; 2; 3) |
|
| SINGLE ARG RULE |
> dd (1,) (1,) > dd [1,] $ = [1] > dd [[1,]] $ = [[1]] |
> dd (1,) (1) > dd [1,] [1] > dd [[1],] [[1],] |
| LIST NOW IMMUTABLE |
> my @array = 1,2,3 1 2 3 > @array.shift 1 > dd @array @array = [2, 3]<> |
> my @list := 1,2,3 (1 2 3) > @list.shift Method 'shift' not found for invocant of class 'List' > @list[0] 1 > dd @list (1, 2, 3) |
| ARRAY IS MUTABLE AND A SUBCLASS OF LIST |
> my @array = 1,2,3 [1 2 3] > @array[0]=0 0 > dd @array @array = [0, 2, 3] >say (Array).^mro ((Array) (List) (Cool) (Any) (Mu)) |
|
| SLIP SUBCLASS OF LIST |
> my @a = 1, (2, 3).Slip, 4 [1 2 3 4] > my $slip = slip(2,3) (2 3) > dd $slip Slip $slip = $(2, 3) > my @array = 1,$slip,4 [1 2 3 4] > (1,$(2,3),4) (1 (2 3) 4) > (1,|(2,3),4) (1 2 3 4) |
|
| SEQUENCE |
> my $grep = (1..4).grep(*>2); dd $grep>>.Int; (3, 4) > dd $grep>>.Int; This Seq has already been iterated, and its values consumed in block prevent consumption> my $grep = (1..4).grep(*>2); my $cache=$grep.cache (3 4) > say $cache>>.Int (3 4) > say $cache>>.Int (3 4) > my @array = 1,(2,3),4 [1 (2 3) 4] > dd @array.flat (1, $(2, 3), 4).Seq > dd @array.flat.list (1, $(2, 3), 4) |
So we are anticipating a long rollout cycle for PHP 6, and we did not want to take the same route that the Perl project did, with project contributors still working on Perl 6 I think six years later. People make fun of Microsoft, but take a look at Perl 6. . . .
Sure, PHP 6 may have a shorter release cycle than Perl 6 has, but at the end of it all, we'll have Perl 6, and you'll still have PHP.So how did those predictions work out? Well, after a little over six years of development, we discovered that we were never going to see a PHP 6 at all. Having seen how long Perl 6 had taken, and how long PHP 6 was taking, the number 6 is associated with failure. So they cancelled PHP 6 and voted to change the name to PHP 7. Problem solved! No, really, this is some of the actual reasoning given by people on the 6 to 7 RFC. (Someone should tell the ES6 folks before the curse strikes our browsers!)
Just sayin'.
xoxo,
Andy
print removed in favor of function print(), ostensibly to make a consistent API but really just to mess with people.At FOSDEM 2015, Larry announced that there will likely be a Perl 6 release candidate in 2015, possibly around the September timeframe. What we’re aiming for is concurrent publication of a language specification that has been implemented and tested in at least one usable compilation environment — i.e., Rakudo Perl 6.
So, for the rest of 2015, we can expect the Rakudo development team to be highly focused on doing only those things needed to prepare for the Perl 6 release later in the year. And, from previous planning and discussion, we know that there are three major areas that need work prior to release: the Great List Refactor (GLR), Native Shaped Arrays (NSA), and Normalization Form Grapheme (NFG).
…which brings us to Parrot. Each of the above items is made significantly more complicated by Rakudo’s ongoing support for Parrot, either because Parrot lacks key features needed for implementation (NSA, NFG) or because a lot of special-case code is being used to maintain adequate performance (lists and GLR).
At present most of the current userbase has switched over to MoarVM as the backend, for a multitude of reasons. And more importantly, there currently aren’t any Rakudo or NQP developers on hand that are eager to tackle these problems for Parrot.
In order to better focus our limited resources on the tasks needed for a Perl 6 language release later in the year, we’re expecting to suspend Rakudo’s support for the Parrot backend sometime shortly after the 2015.02 release.
Unfortunately the changes that need to be made, especially for the GLR, make it impractical to simply leave existing Parrot support in place and have it continue to work at a “degraded” level. Many of the underlying assumptions will be changing. It will instead be more effective to (re)build the new systems without Parrot support and then re-establish Parrot as if it is a new backend VM for Rakudo, following the techniques that were used to create JVM, MoarVM, and other backends for Rakudo.
NQP will continue to support Parrot as before; none of the Rakudo refactorings require any changes to NQP.
If there are people that want to work on refactoring Rakudo’s support for Parrot so that it’s more consistent with the other VMs, we can certainly point them in the right direction. For the GLR this will mainly consists of migrating parrot-specific code from Rakudo into NQP’s APIs. For the NSA and NFG work, it will involve developing a lot of new code and feature capabilities that Parrot doesn’t possess.
This past weekend I attended the 2014 Austrian Perl Workshop and Hackathon in Salzburg, which turned out to be an excellent way for me to catch up on recent changes to Perl 6 and Rakudo. I also wanted to participate directly in discussions about the Great List Refactor, which has been a longstanding topic in Rakudo development.
What exactly is the “Great List Refactor” (GLR)? For several years Rakudo developers and users have identified a number of problems with the existing implementation of list types — most notably performance. But we’ve also observed the need for user-facing changes in the design, especially in generating and flattening lists. So the term GLR now encompasses all of the list-related changes that seem to want to be made.
It’s a significant (“great”) refactor because our past experience has shown that small changes in the list implementation often have far-reaching effects. Almost any bit of rework of list fundamentals requires a fairly significant refactor throughout much of the codebase. This is because lists are so fundamental to how Perl 6 works internally, just like the object model. So, as the number of things that are desirable to fix or change has grown, so has the estimated size of the GLR effort, and the need to try to achieve it “all at once” rather than piecemeal.
The pressure to make progress on the GLR has been steadily increasing, and APW2014 was significant in that a lot of the key people needed for that would be in the same location. Everyone I’ve talked to agrees that APW2014 was a smashing success, and I believe that we’ve now resolved most of the remaining GLR design issues. The rest of this post will describe that.
This is an appropriate moment to recognize and thank the people behind the APW effort. The organizers did a great job. The Techno-Z and ncm.at venues were fantastic locations for our meetings and discussions, and I especially thank ncm.at, Techno-Z, yesterdigital, and vienna.pm for their generous support in providing venues and food at the event.
So, here’s my summary of GLR issues where we were able to reach significant progress and consensus.
(Be sure to visit our gift shop!)
Much of the GLR discussion at APW2014 concerned flattening list context in Perl 6. Over the past few months and years Perl 6 has slowly but steadily reduced the number of functions and operators that flatten by default. In fact, a very recent (and profound) change occurred within the last couple of months, when the .[] subscript operator for Parcels switched from flattening to non-flattening. To illustrate the difference, the expression
(10,(11,12,13),(14,15)).[2]
previously would flatten out the elements to return 12, but now no longer flattens and produces (14,15). As a related consequence, .elems no longer flattens either, changing from 6 to 3.
Unfortunately, this change created a inconsistency between Parcels and Lists, because .[] and .elems on Lists continued to flatten. Since programmers often don’t know (or care) when they’re working with a Parcel or a List, the inconsistency was becoming a significant pain point. Other inconsistencies were increasing as well: some methods like .sort, .pick, and .roll have become non-flattening, while other methods like .map, .grep, and .max continue to flatten. There’s been no really good guideline to know or decide which should do which.
Flattening behavior is great when you want it, which is a lot of the time. After all, that’s what Perl 5 does, and it’s a pretty popular language. But once a list is flattened it’s hard to get the original structure if you wanted that — flattening discards information.
So, after many animated discussions, review of lots of code snippets, and seeking some level of consistency, the consensus on Perl 6 flattening behavior seems to be:
[ ] array constructor are unchanged; they continue to flatten their input elements. (Arrays are naturally flat.)for @a,@b { ... } flattens @a,@b and applies the block to each element of @a followed by each element of @b. Note that flattening can easily be suppressed by itemization, thus for @a, $@b { ... } flattens @a but does all of @b in a single iteration..map, .grep, and .first… the programmer will have to use .flat.grep and .flat.first to flatten the list invocant. Notably, .map will no longer flatten its invocant — a significant change — but we’re introducing .for as a shortcut for .flat.map to preserve a direct isomorphism with the for statement.There’s ongoing conjecture of creating an operator or syntax for flattening, likely a postfix of some sort, so that something like .|grep would be a convenient alternative to .flat.grep, but it doesn’t appear that decision needs to be made as part of the GLR itself.((1,2), 3, (4,5)).map({...}) # iterates over three elements
map {...}, ((1,2),3,(4,5)) # iterates over five elements
(@a, @b, @c).pick(1) # picks one of three arrays
pick 1, @a, @b, @c # flatten arrays and pick one element
As a result of improvements in flattening consistency and behavior, it appears that we can eliminate the Parcel type altogether. There was almost unanimous agreement and enthusiasm at this notion, as having both the Parcel and List types is quite confusing.
Parcel was originally conceived for Perl 6 as a “hidden type” that programmers would rarely encounter, but it didn’t work out that way in practice. It’s nice that we may be able to hide it again — by eliminating it altogether. 
Thus infix:<,> will now create Lists directly. It’s likely that comma-Lists will be immutable, at least in the initial implementation. Later we may relax that restriction, although immutability also provides some optimization benefits, and Jonathan points out that may help to implement fixed-size Arrays.
Speaking of optimization, eliminating Parcel may be a big boost to performance, since Rakudo currently does a fair bit of converting Parcels to Lists and vice-versa, much of which goes away if everything is a List.
During a dinner discussion Jonathan reminded me that Synopsis 4 has all of the looping constructs as list generators, but Rakudo really only implements for at the moment. He also pointed out that if the loop generators are implemented, many functions that currently use gather/take could potentially use a loop instead, and this could be much more performant. After thinking on it a bit, I think Jonathan is on to something. For example, the code for IO::Handle.lines() currently does something like:
gather {
until not $!PIO.eof {
$!ins = $!ins + 1;
take self.get;
}
}
With a lazy while generator, it could be written as
(while not $!PIO.eof { $!ins++; self.get });
This is lazily processed, but doesn’t involve any of the exception or continuation handling that gather/take requires. And since while might choose to not be strictly lazy, but lines() definitely should be, we may also use the lazy statement prefix:
lazy while not $!PIO.eof { $!ins++; self.get };
The lazy prefix tells the list returned from the while that it’s to generate as lazily as it possibly can, only returning the minimum number of elements needed to satisfy each request.
So as part of the GLR, we’ll implement the lazy list forms of all of the looping constructs (for, while, until, repeat, loop). In the process I also plan to unify them under a single LoopIter type, which can avoid repetition and be heavily optimized.
This new loop iterator pattern should also make it possible to improve performance of for statements when performed in sink context. Currently for statements always generate calls to .map, passing the body of the loop as a closure. But in sink context the block of a for statement could potentially be inlined. This is the way blocks in most other loops are currently generated. Inlining the block of the body could greatly increase performance of for loops in sink context (which are quite common).
Many people are aware of the problem that constructs such as for and map aren’t “consuming” their input during processing. In other words, if you’re doing .map on a temporary list containing a million elements, the entire list stays around until all have been processed, which could eat up a lot of memory.
Naive solutions to this problem just don’t work — they carry lots of nasty side effects related to binding that led us to design immutable Iterators. We reviewed a few of them at the hackathon, and came back to the immutable Iterator we have now as the correct one. Part of the problem is that the current implementation is a little “leaky”, so that references to temporary objects hang around longer than we’d like and these keep the “processed” elements alive. The new implementation will plug some of the leaks, and then some judicious management of temporaries ought to take care of the rest.
In the past year much work has been done to improve sink context to Rakudo, but I’ve never felt the implementation we have now is what we really want. For one, the current approach bloats the codegen by adding a call to .sink after every sink-context statement (i.e., most of them). Also, this only handles sink for the object returned by a Routine — the Routine itself has no way of knowing it’s being called in sink context such that it could optimize what it produces (and not bother to calculate or return a result).
We’d really like each Routine to know when it’s being called in sink context. Perl 5 folks will instantly say “Hey, that’s wantarray!”, which we long ago determined isn’t generally feasible in Perl 6.
However, although a generalized wantarray is still out of reach, we can provide it for the limited case of detecting sink contexts that we’re generating now, since those are all statically determined. This means a Routine can check if it’s been called in sink context, and use that to select a different codepath or result. Jonathan speculates that the mechanism will be a flag in the callsite, and I further speculate the Routine will have a macro-like keyword to check that flag.
Even with detecting context, we still want any objects returned by a Routine to have .sink invoked on them. Instead of generating code for this after each sink-level statement, we can do it as part of the general return handler for Routines; a Routine in sink context invokes .sink on the object it would’ve otherwise returned to the caller. This directly leads to other potential optimizations: we can avoid .sink on some objects altogether by checking their type, and the return handler probably doesn’t need to do any decontainerizing on the return value.
As happy as I am to have discovered this way to pass sink context down into Routines, please don’t take this as opening an easy path to lots of other wantarray-like capabilities in Perl 6. There may be others, and we can look for them, but I believe sink context’s static nature (as well as the fact that a false negative generally isn’t harmful) makes it quite a special case.
One area that has always been ambiguous in the Synopses is determining when various contextualizing methods must return a copy or are allowed to return self. For example, if I invoke .values on a List object, can I just return self, or must I return a clone that can be modified without affecting the original? What about .list and .flat on an already-flattened list?
The ultra-safe answer here is probably to always return a copy… but that can leave us with a lot of (intermediate) copies being made and lying around. Always returning self leads to unwanted action-at-a-distance bugs.
After discussion with Larry and Jonathan, I’ve decided that true contextualizers like .list and .flat are allowed to return self, but other method are generally obligated to return an independent object. This seems to work well for all of the methods I’ve considered thus far, and may be a general pattern that extends to contextualizers outside of the GLR.
(small matter of programming and documentation)
The synopses — especially Synopsis 7 — have always been problematic in describing how lists work in Perl 6. The details given for lists have often been conjectural ideas that quickly prove to epic fail in practice. The last major list implementation was done in Summer 2010, and Synopsis 7 was supposed to be updated to reflect this design. However, the ongoing inconsistencies (that have led to the GLR) really precluded any meaningful update to the synopses.
With the progress recently made at APW2014, I’m really comfortable about where the Great List Refactor is leading us. It won’t be a trivial effort; there will be significant rewrite and refactor of the current Rakudo codebase, most of which will have to be done in a branch. And of course we’ll have to do a lot of testing, not only of the Perl 6 test suite but also the impact on the module ecosystem. But now that much of the hard decisions have been made, we have a roadmap that I hope will enable most of the GLR to be complete and documented in the synopses by Thanksgiving 2014.
Stay tuned.
"I'm just happy that the two of you liked my work." -- vanstynAlthough he was talking about DBIx, I think that captures the spirit of conference as a whole. All of us here -- from the n00bs to the pumpkings -- want to share our work and make something useful for others. It's not an organization where we wait for pronouncements from on high, but one where users create endless variations and share them. Not an organization so much as a family.
"We have faith, hope, and love, but the most awesome of these is love." -- Larry WallA line like this might seem a bit hokey out of context, but it was actually moving when I heard it. We have faith that we can use Perl to solve our problems. We have hope that Perl 5 and 6 will continue to get better. And we love Perl, unconditionally, despite all of her flaws. And as Wil Wheaton says about us geeks, we just want to love our special thing the best we can, and go the extra mile to share it with others.
[This is a response to the Russian Perl Podcast transcribed by Peter Rabbitson and discussed at blogs.perl.org.]
I found this translation and podcast to be interesting and useful, thanks to all who put it together.
Since there seems to have been some disappointment that Perl 6 developers didn’t join in the discussions about “Perl 7” earlier this year, and in the podcast I’m specifically mentioned by name, I thought I’d go ahead and comment now and try to improve the record a bit.
While I can’t speak for the other Perl 6 developers, in my case I didn’t contribute to the discussion because nearly all the things I would’ve said were already being said better by others such as Larry, rjbs, mst, chromatic, etc. I think a “Perl 7” rebrand is the wrong approach, for exactly the reasons they give.
A couple of statements in the podcast refer to “hurting the feelings of Perl 6 developers” as being a problem resulting from a rebrand to Perl 7. I greatly appreciate that people are concerned with the possible impact of a Perl 5 rebrand on Perl 6 developers and our progress. I believe that Perl 6’s success or failure at this point will have little to do with the fact that “6 is larger than 5”. I don’t find the basic notion of “Perl 7” offensive or directly threatening to Perl 6.
But I fully agree with mst that “you can’t … have two successive numbers in two brands and not expect people to be confused.” We already have problems explaining “5” and “6” — adding more small integers to the explanation would just make an existing problem even worse, and wouldn’t do anything to address the fundamental problems Perl 6 was intended to resolve.
Since respected voices in the community were already saying the things I thought about the name “Perl 7”, I felt that adding my voice to that chorus could only be more distracting than helpful to the discussion. My involvement would inject speculations on the motivations of Perl 6 developers into what is properly a discussion about how to promote progress with Perl 5. I suspect that other Perl 6 developers independently arrived at similar conclusions and kept silent as well (Larry being a notable exception).
I’d also like to remark on a couple of @sharifulin’s comments in the podcast (acknowledging that the transcribed comments may be imprecise in the translation from Russian):
First, I’m absolutely not the “sole developer” of Perl 6 (13:23 in the podcast), or even the sole developer of Rakudo Perl 6. Frankly I think it’s hugely disrespectful to so flippantly ignore the contributions of others in the Perl 6 development community. Let’s put some actual facts into this discussion… in the past twelve months there have been over 6,500 commits from over 70 committers to the various Perl 6 related repositories (excluding module repositories), less than 4% (218) of those commits are from me. Take a look at the author lists from the Perl 6 commit logs and you may be a little surprised at some of the people you find listed there.
Second, there is not any sense in which I think that clicking “Like” on a Facebook posting could be considered “admitting defeat” (13:39 in the podcast). For one, my “Like” was actually liking rjbs’ reply to mst’s proposal, as correctly noted in the footnotes (thanks Peter!).
But more importantly, I just don’t believe that Perl 5 and Perl 6 are in a battle that requires there to be a conquerer, a vanquished, or an admission of defeat.
Pm
$foo->WHAT can tell you if you have a Str, Int, or IO::Handle. $path =~ s/^([a-z]:)/\l$1/s;//server/share) that OS2.pm had only half-implemented. And so a huge block of code cruft bit the dust.sub _tmpdir {
my $self = shift;
my @dirlist = @_;
my $tmpdir;
foreach (@dirlist) {
next unless defined && -d && -w _;
$tmpdir = $_;
last;
}
return $self->canonpath($tmpdir);
}$_, @_, and shift.method !tmpdir( *@dirlist ) {
my $tmpdir = first { .defined && .IO.w && .IO.d }, @dirlist;
return self.canonpath($tmpdir);
}$tmpdir to the first defined writable directory in @dirlist." Less, easier to read code is easier to maintain.if( $foo ) to if $foo, etc.git, and make -- enough to commit to repositories and build a software package, anyway.git clone it to on your own machine.rakudo directory. There are a few setup things that you'll want to do. First of all, go ahead and build Rakudo, using the normal steps: perl ./Configure.pl --gen-parrot
make
make install$PATH environment variable. Which, if you don't know how to do it -- well here's Google. In particular, you'll need to add the full path to the rakudo/install/bin directory. make spectestt/spec before hitting ^C. You will need these tests later to make sure you didn't break anything. git remote add upstream git://github.com/rakudo/rakudo.git git clone git://github.com/tadzik/panda.git
cd panda
perl6 bootstrap.pl git checkout -b mynewbranchnamerakudo/src folder, so this is where you'll want to edit the contents.vm directory contains files specific to the virtual machines Rakudo runs under. At this time of this writing, there's only one thing in there, parrot, but very soon there will also be a jvm directory. Exciting! Most of the purpose of this code is to map functions to lower-level operations, in either Parrot or Java.Perl6 directory contains the grammar and actions used to build the language, as well as the object metamodel. The contents of this folder are written in NQP, or Not Quite Perl. This section determines how the language is parsed.core directory contains the files that will be built into the core setting. You'll find classes or subroutines in here for just about everything in Perl: strings, operators like eq, filehandles, sets, and more. Individual files look similar to modules, but these are "modules" that are available to every Perl 6 program.gen directory contains files that are created in the compliation process. The core setting lives here, creatively named CORE.setting. And if you look at it, it's just a concatenation of the files in core, put together in the order specified in rakudo/tools/build/Makefile.in. While these files can and do get overwritten in the build process, it's often a good idea to keep a copy of CORE.setting open so you can find what you're looking for faster -- and then go edit it in core.git bisect for problems later. And push your edits to Github as a free backup. If you get stuck, drop by #perl6 on irc.freenode.net and ask questions. git fetch upstream
git merge upstream/nom
perl Configure.pl
make
make spectest #?pugs 1 skip 'reason'
#?niecza 1 skip 'reason'rakudo/t/spectest.data. If your code fixes broken tests, then you'll want to *unfudge* by removing the #?rakudo skip lines above the relevant tests. perl6 panda/rebootstrap.plgit commit; git push will add it to the ticket. If there aren't any problems, someone will just merge it in a couple days.At YAPC::NA 2012 in Madison, WI I gave a lightning talk about basic improvements in Rakudo’s performance over the past couple of years. Earlier today the video of the lightning talks session appeared on YouTube; I’ve clipped out my talk from the session into a separate video below. Enjoy!
A couple of weeks ago I entered the Dallas Personal Robotics Group Roborama 2012a competition, and managed to come away with first place in the RoboColumbus event and Line Following event (Senior Level). For my robot I used one of the LEGO Mindstorms sets that we’ve been acquiring for use by our First Lego League team, along with various 3rd party sensors.
The goal of the RoboColumbus event was to build a robot that could navigate from a starting point to an ending point placed as far apart as possible; robots are scored on distance to the target when the robot stops. If multiple robots touch the finish marker (i.e., distance zero), then the time needed to complete the course determines the rankings. This year’s event was in a long hall with the target marked by an orange traffic cone.
HiTechnic IR ball and IRSeeker sensor
Contestants are allowed to make minor modifications to the course to aid navigation, so I equipped my robot with a HiTechnic IRSeeker sensor and put an infrared (IR) electronic ball on top of the traffic cone. The IRSeeker sensor reports the relative direction to the ball (in multiples of 30 degrees), so the robot simply traveled forward until the sensor picked up the IR signal, then used the IR to home in on the traffic cone. You can see the results of the winning run in the video below, especially around the 0:33 mark when the robot makes its first significant IR correction:
http://youtu.be/x1GvpYAArfY
My first two runs of RoboColumbus didn’t do nearly as well; the robot kept curving to the right for a variety of reasons, and so it never got a lock on the IR ball. Some quick program changes at the contest and adjustments to the starting direction finally made for the winning run.
For the Line Following contest, the course consisted of white vinyl tiles with electrical tape in various patterns, including line gaps and sharp angles. I used a LineLeader sensor from mindsensors.com for basic line following, with some heuristics for handling the gap conditions. The robot performed fine on my test tiles at home, but had difficulty with the “gap S curve” tiles used at the contest. However, my robot was the only one that successfully navigated the right angle turns, so I still ended up with first place.
Matthew and Anthony from our FLL robotics team also won other events in the contest, and there are more videos and photos available. The contest was a huge amount of fun and I’m already working on new robot designs for the next competition.
Many thanks to DPRG and the contest sponsors for putting on a great competition!