JavaScript REPL for Windows: Part 3–Dynamic Breakpoints

This is part three of three part post about JavaScript REPL console for Windows. In part one I have explained motivation and some initial steps, in part two I have added debug REPL and static breakpoints. In part three I am going to add the ability to set breakpoints dynamically from within JSREPL.

As mentioned previously, full source code of JSREPL is freely available under GPL 2 license on github.

Dynamic Breakpoints

While static breakpoints offer good benefits of being flexible, they are, well, static. When I am debugging a script it is hard to predict where all the breakpoints would have to be placed. So, if I decide that I need another breakpoint in the middle of the debugging session, I need to edit the source code and restart the session.

It would be nice if I could set the breakpoints on the fly without the need to restart the session. It turns out, there is an easy way to do that for functions declared at the global scope; either global functions or anonymous functions referenced by global variables. Furthermore, it is even possible to do this for anonymous functions declared within such global functions, for example, object methods. There are some caveats to the latter part, though.

Well then, my JSREPL is getting three new commands:

• bp – for adding new breakpoints
• bd – for deleting breakpoints
• bl – for listing breakpoints 

  The logic of adding a new breakpoint as well as removing one is quite simple – just figure out all the lines that the function needs to have breakpoints and generate a new string with source code for the function. Eval that at the global scope and subsequent invocations of the function will trigger breakpoints.

  There is one fine point in all of this: whether function is anonymous or not. But that is easily determined by the parseFunction  method from part two. If the function is anonymous, then the eval should be an assignment to a global variable. Otherwise, it is just a function declaration.

  Figure 14. Method to update breakpoints.

  1     updateFnBreaks: function (bpInfo, bpLine) {
  2         var lineNos = Arr(bpInfo.breaks).map(function(eBr) { return eBr.line; }).a;
  3         if (typeof(bpLine) != “undefined”) {
  4             lineNos.push(bpLine);
  5         }
  6         lineNos.sort();
  7
  8         var fnLines = bpInfo.savedFunc.slice(0);    // copy the array
  9         var bpLabelPrefix = bpInfo.name + “(“;
  10         for (iLN in lineNos) {
  11             var offset = lineNos[iLN] + parseInt(iLN);    // compensating for lines inserted already
  12             if (offset < fnLines.length) {
  13                 var bpLabel = bpInfo.name + “(“ + lineNos[iLN].toString() + “)”;
  14                 fnLines.splice(offset, 0 /*=toDelete*/, “eval(dbgBreak(\”“ + bpLabel + “\”)) // <dbg_break>”);
  15             } else {
  16                 println(“ERROR: line “, lineNos[iLN], ” is outside of the function”);
  17                 return null;
  18             }
  19         }
  20         return fnLines.join(“\n\r“);
  21     }
  

   

  To illustrate the dynamic nature of the breakpoints let’s look at another example. This time we are dealing with class method.

  Figure 15. Source code for the example of dynamic breakpoints.

  1 classA = function (n) {
  2     return {
  3         name:  n,
  4         who:  function () {
  5             println(“Name=”, this.name);
  6             return this.name;
  7         }
  8     };
  9 };
  

   

  In this case I would like to set a breakpoint on line 6, but that is in the anonymous function, which happens to be a class method. Here is how that worked out for me:

  Figure 16. Session log of dynamic breakpoint demo.

  1 # list classA
  2    0 function (n) {
  3    1     return {
  4    2         name:  n,
  5    3         who:  function () {
  6    4             println(“Name=”, this.name);
  7    5             return this.name;
  8    6         }
  9    7     };
  10    8 }
  11 # bp classA 5
  12 # var a1 = new classA(“John”)
  13
  14 # a1.who()
  15 Name=John
  16
  17 dbg break – “classA(5)”  ? – displays list of debugger commands.
  18 dbg> w
  19 Call stack:
  20
  21 Level Function
  22 ~~~~~ ~~~~~~~~~~~~
  23     0 <anonymous>()
  24 dbg> a1.name
  25 John
  26 dbg> g
  27 John
  28 # bl
  29 Active breakpoints:
  30
  31 Id  Function(line)
  32 ~~~~ ~~~~~~~~~~~~~~
  33    0 classA(5)
  34 # bd 0
  35 # bl
  36 Active breakpoints:
  37
  38 Id  Function(line)
  39 ~~~~ ~~~~~~~~~~~~~~
  40 # a1.who()
  41 Name=John
  42
  43 dbg break – “classA(5)”  ? – displays list of debugger commands.
  44 dbg> g
  45 John
  46 # var a2 = new classA(“Peter”)
  47
  48 # a2.who()
  49 Name=Peter
  50 Peter
  51 #
  

  Setting the breakpoint (listing line 7) triggers the break into debug REPL (listing line 17). However, removing the breakpoint (listing line 43) does not remove the breakpoint from the method of object a1 (listing lines 40 – 44). However, creating a new instance of classA after removing the breakpoint acts as expected (listing line 48).

  This is it. While there are some caveats to dynamic breakpoints, it is still a very useful addition to JSREPL.

  Next Steps

  JSREPL is a work in progress and will be for quite some time. I am refining the functionality and addressing more corner cases as I run into those. If you use JSREPL and run into issues, please report them on the github issues page.

  I already have plans for the future – I would like to add enabled/disabled flag to the breakpoint, such that the  breakpoints that cannot be deleted, could at least be disabled. I would like to add an indicator of the current breakpoint in the function source listing. The wiki on github needs updating as well.

  Thank you for reading and happy debugging.

JavaScript REPL for Windows: Part 2–Breakpoints and Debug REPL

This is part two of a three part post about JavaScript REPL console for Windows. In part one I have explained motivation and some initial steps to get the interesting part – how make JavaScript to debug itself.

As mentioned previously, full source code of JSREPL is freely available under GPL 2 license on github.

Breakpoints

To me, the most essential capability of a debugger is to stop execution of the code in an arbitrary place and inspect the state of the program. Stepping through code, callstack, etc. is all secondary to breakpoints. For this reason I have focused on getting the ability to stop the execution first.

The way to stop the execution is quite simple – just call a function (let’s name it ‘inner function), which runs REPL and does all the state inspections inside the REPL. However, due to the way JavaScript variable binding works, inside that inner function, variables can only bind to its scope and enclosing scopes through eval. One way to solve this problem is to declare a function inside the function I am trying to debug, but doing so by hand is quite tedious – I would have to do that every time I want to place a breakpoint.

The solution is to call the breakpoint function through eval, which in turn declares an anonymous utility function to evaluate expressions in the scope of the function being debugged.

Figure 6. Breakpoint function.

1 function dbgBreak(msg) {
2     return "DBG.repl(\"" + msg.toString() +
3            "\" , function (v) { return eval(v); }, " +
4            "typeof arguments != \"undefined" && arguments != null ? arguments.callee : null)";
5 }
This code returns a string, which when evaluated triggers a breakpoint. The breakpoint runs debug REPL (line two). There are three parameters passed onto debug REPL – a breakpoint label (so that I know which breakpoint triggered), an anonymous function, which evaluates expressions in the scope of the function being debugged, and a reference to the function object, if present.

The breakpoint in the source code looks like this:

Figure 7. Setting a static breakpoint.

1 eval(dbgBreak("BP1"));

When the the following code is executed, it breaks into debug REPL.

Figure 8. Sample script to illustrate breakpoint.

1 function A(param) {
2     println("param=", param);
3     eval(dbgBreak("BP1"));
4     return param + 1;
5 }

Figure 9. Example of breakpoint.

1 Interactive JavaScript REPL console. Version  1.1
2
3 When no parameters are passed on command line, JSREPL starts
4 in interactive mode with the following commands available:
5     println(<expr>[, …]) – prints expressions to stdout with \n.
6     print(<expr>[,…])    – prints expressions to stdout without \n.
7     \ at the end of line   – indicates continuation for multiline input.
8     load <file>            – loads and executes file in current session.
9     list <fn>              – list the source of the function. <fn> must
10                            – be function name without parentheses.
11     bp <fn> <line>         – inserts a breakpoint in function <fn> before
12                              line <line>. Lines are zero-based.
13     bd <id>                – removes breakpoint with id <id>.
14     bl                     – lists all breakpoints.
15     quit                   – terminates execution.
16
17 When a file name is passed as a first parameter on a command line,
18 it is executed in the context of this console and all arguments
19 are passed on to the script as is.
20
21 # load ..\bptest.js
22
23 # A(1)
24 param=1
25
26 dbg break – "BP1"  ? – displays list of debugger commands.
27 dbg>

 

The commands and expressions I typed are highlighted in blue. At this point we have a way to interrupt the execution, but we still need to complete debug REPL.

Debug REPL

Debug REPL has to be able to inspect the state of a program (we already made provisions for that in our breakpoint implementation). It also needs to be display a call stack, list function source and resume execution. Listing function source is reusing the same code used in JavaScript REPL, described in part one, while resuming execution is simply returning from debug REPL. That leaves us with the call stack.

The third parameter passed onto the DBG.repl method is a function object. We can easily navigate the call stack using its caller property.

Figure 10. Stack walking function.

1 walkStack: function (fn) {
2     var stack = new Array();
3     while (fn != null) {
4         stack.push(this.parseFunction(fn));
5         fn = fn.caller;
6     }
7     return stack;
8 }

As before this is a method of DBG object. The parseFunction() method does a little trickery extracting the function name from the source of the function, as well as determines formal parameter names, then matches them with actual parameter values.

.Figure 11. Parsing function source code,

1 parseFunction: function (fn) {
2     var fnText = fn;
3     if (typeof(fn) == "function") {
4         fnText = fn.toString();
5     }
6     var res = "<Global>";
7     // get rid of comments and newlines:
8     fnText = fnText.replace(/\/\/.*(?=\r\n|\n\r|$)/g,"");
9     fnText = fnText.replace(/\/\*(?:\*(?!\/)|[^*])*\*\//g,"");
10     fnText = fnText.replace(/\r\n|\n\r/g, "");
11    
12     // find function name
13     var r = fnText.match(/^\s*function(?:\s+(\w*)|)\s*\(/)
14     if (r != null) {
15         res = RegExp.$1 == "" ? "<anonymous>" : RegExp.$1.toString();
16     }
17    
18     // find arguments
19     var reParamName = /\s*(\w*|"(?:\\"|[^"])*")\s*(,|\))/;
20     var params = new Array();
21     var lastIndex = RegExp.lastIndex;
22     while (lastIndex >= 0 && (r = fnText.substr(lastIndex).match(reParamName)) != null) {
23         lastIndex = RegExp.lastIndex != -1 ? lastIndex + RegExp.lastIndex : -1;
24         if (RegExp.$1.length > 0) {
25             params.push(RegExp.$1);
26         }
27         if (RegExp.$2 == ")") { // end of parameters
28             break;
29         }
30     }
31     return { name: res, params: params, func: fn};
32 }

 

Let’s see the code in action! I am using the following sample script to set the breakpoint.

Figure 12. Sample script to illustrate debug REPL.

1 function A(param) {
2     println("param=", param);
3     eval(dbgBreak("BP1"));
4     return param + 1;
5 }
6
7 function B(p1, p2, p3) {
8     return A(p2, p1, 20);
9 }
10
11 function C() {
12     return B("Text", 10);
13 }
14

When executed,  debug REPL starts.

Figure 13. Debug REPL session log.

1 # load ..\bptest.js
2
3 # C(1, "a", null)
4 param=10
5
6 dbg break – "BP1"  ? – displays list of debugger commands.
7 dbg> ?
8
9 dbg commands:
10
11 g          – resume execution.
12 w          – print call stack.
13 l <fun>    – print source of function <fun>.
14              <fun> is either function name or
15              stack level (as displayed by w).
16 bl         – list all breakpoints.
17 q          – quit execution of the script.
18 <expr>     – evaluate <expr> in the context of
19              the current function.n
20 dbg> w
21 Call stack:
22
23 Level Function
24 ~~~~~ ~~~~~~~~~~~~
25     0 A(param=10, "Text", 20)
26     1 B(p1="Text", p2=10, p3=)
27     2 C(1, "a", null)
28 dbg> l 2
29    0 function C() {
30    1     return B("Text", 10);
31    2 }
32 dbg> l 1
33    0 function B(p1, p2, p3) {
34    1     return A(p2, p1, 20);
35    2 }
36 dbg> l 0
37    0 function A(param) {
38    1     println("param=", param);
39    2     eval(dbgBreak("BP1"));
40    3     return param + 1;
41    4 }
42 dbg> param
43 10
44 dbg> param=20
45 20
46 dbg> g
47 21
48 #
49

 

At this point the debugger has quite a bit of functionality, and while static breakpoints I have to set in my source code might look like tedious, the mechanism is also very powerful – I can set conditional breakpoints with ease.

Now if only I could set the breakpoints dynamically… Stay tuned for the next part.

JavaScript REPL for Windows: Part1–Motivation, Choices and First Steps

I earn a living writing C++ code. Code that twiddles pixels, mostly, and, while I write the occasional web app page, I am not a web application developer. Yet I am a avid user of JavaScript.

Long before node.js rose to its popularity Microsoft Windows users had access to all the power of a scripting engine on a command line– cscript.exe. It is a very convenient and powerful language+host combination to develop simple utility scripts. Much faster than writing CLI (Command Line Interface) C++ utility. CScript.exe is on all modern flavors of Windows, thus it is as portable (Windows-vise) as an equivalent C++ exe.

This post is a story in three parts of how I ended up implementing a JavaScript REPL console with simple debugger REPL and breakpoints. In this first part I am going to explain how I arrived at the decision to write JSREPL and how it all started. Full source code of JSREPL is freely available under GPL 2 license on github. If you’d rather not follow my journey, you can jump directly there.

Motivation

Visual Studio is the default development and debugging tool for JavaScript, when working with cscript.exe. While it is a very powerful and full featured debugger, rapid development and debugging is hindered by two facts – first, there is no support for cscript.exe JavaScript projects in Visual Studio.

Second, the just-in-time debugging via //X command line option is cumbersome. It pops a pop-under dialog where you need to select a version and instance of visual studio to use for debugging every time a new debugging session is started. And what about that startup time of Visual Studio 2010?

To me, the power of JavaScript is in its immediate zero-compile-time nature. I would like to be able to move fast, in a rapid edit-run-fix succession. Such desire left me searching the internet for an alternative JavaScript debugger. There are several mentions of an older Microsoft Script Debugger, which is not as feature rich as Visual Studio and is not supported on newer versions of Windows, but nothing else along the lines of my hopes of simple REPL (read-evaluate-print-loop) style debugger.

Thus I decided to write one myself.

Choices

I had written a trivial JavaScript REPL script a few years ago (see below) and I wanted to continue in a similar direction – command line interface, simple quick and reuse as much of JavaScript engines capability as possible.

Figure 1. An example of my early JavaScript REPL.

1 WScript.StdOut.Write("# ");
2 while ((cmdLine = WScript.StdIn.ReadLine()).toLowerCase() != "quit" ) {
3     try {
4          WScript.Echo("\n",(eval(cmdLine)));
5     } catch (err) {
6         WScript.Echo("\n", err.toString());
7     }
8     WScript.StdOut.Write("# ");
9 }

 

The early REPL was useful to me in many ways – from debugging regular expressions to analyzing COM object API behavior. As the time went by, I have added more features – multiline mode (line continuation character) and some global utility functions (print, println, string prototypes, etc.) The script became more useful, but it was still severely lacking a debugging facility —at the very least, some ability to interrupt execution of the program and inspect variables.

So I had to write the debugger. Looking at the Windows Script Runtime documentation on MSDN, one choice became evident – write a full scale debugger in C++ using Script Runtime Hosting API. Even though I write C++ for living and COM is flowing in my blood, I was not too excited to write a full scale host for the Scripting Runtime, then have to register it on the target machine, etc. I decided to look for a simpler solution.

The simpler solution was to evolve my early REPL.

First Steps

Since I had a quite useful REPL going, I wanted to extend it rather than start from scratch. The inventory of the new functionality came down to the following:

1. Facility to load script files into REPL – a load command.
2. Facility to list source code of a function – a list command.
3. Facility to set breakpoints and a debug REPL.

The load command seemed simple enough:

Figure 2. Simple function to load script from file.

1 function load(filePath) {
2     var fso = new ActiveXObject("Scripting.FileSystemObject");
3     if (fso.FileExists(filePath)) {
4         return fso.OpenTextFile(filePath).ReadAll();
5     }
6     return null;
7 }
8

Then, just eval the returned text inside the main loop to create all the objects in the global scope:

Figure 3. Loading of a script via eval.

1 eval(load("Test1.js"));

 

The load function could be further enhanced by searching the JSREPL folder and current PATH to load the file. An added benefit of the load function is that now I can easily break my REPL implementation into smaller files and not bother with WSF format.

A list command did not seem too hard either – just obtain a reference to function object, then do toString() on it to obtain the string representation, split on newline, etc. If you are following along the source code, this facility is in dbg.js – by this time my JSREPL project has three files – js.cmd (more on .cmd part later), dbg.js and util.js.

Figure 4. Function to display source code.

1 listFunction: function (fn) {
2     var fnText =  fn.toString().split(/\n\r|\r\n/);
3     for (l in fnText) {
4             println(l.rJustify(4), " ", fnText[l]);
5     }
6 }

In the above code there are couple of utility functions – rJustify and println. Both are in util.js. The function is a method of the DBG class, hence the syntax.

The js.cmd is a JavaScript file, but it is executed by cmd.exe first, such that cscript.exe is invoked with specific command line options. This is done via some trickery:

Figure 5. Cmd to cscript bootstrap code.

1 @set @a=0 /*
2 @set @a=
3 @%js32path%cscript //E:JScript //I //Nologo %~dpnx0 %*
4 @goto :EOF */

 

This code needs a bit of explanation. It is not my invention – I saw it somewhere circa 2004 – and it is very clever. Some scripting interpreters, such as Python or Perl, have a special command line option to skip one or more lines of source file. This allows to have batch file (shell script) invoke itself via the interpreter. Cscript.exe offers no such  option. So the first line of the code serves purpose of fooling cscript.exe into the JavaScript comment, while at the same time keeping cmd.exe quiet about it.

Cmd.exe reads the first line as “quietly assign 0 /* to variable  @a”, while cscript.exe reads the line as “conditional compilation variable @set, followed by assignment to conditional compilation variable @a, then followed by a JavaScript  comment”.

Lines 2 through 4 are ignored by cscript.exe, since that is JavaScript comment. However, cmd.exe reads line two as “quietly unassign variable @a. Then line three “quietly execute cscript.exe passing this file as first parameter”, and, finally, line four tells cmd.exe to “quietly exit”. Pretty neat.

Let’s take an inventory here. By now, JSREPL is quite a useful interactive environment – I can load scripts and even see exactly where I want to place the breakpoints. In the part two I will break down the break points (pun?) and debug REPL.