archaelus

mood: excited

I've had one of these “And _this_ is why you want erlang” moments.

I have a system under test which it's not convenient to recompile right now (I'm doing exploratory experimental surgery on the codebase and not using git as much as I should). I finally have the client connected to it and it starts crashing. Hmm. Read the OTP logs...

(erms@testbox)> rb:rescan([{max, 20}]), rb:show().

...

exited: {undef,[{erms_smpp,pdu_to_msg, [...]},...]}

...

Right. Missing function. Could have sworn I wrote that. Check dev box. Yup - sure did. Left it out of the deployment. Boh. MaHa!

devbox $ scp ebin/erms_smpp.beam testbox:

(erms@testbox)> code:load_abs(”/home/me/erms_smpp”).

Hot code loading saves my bacon once again.

Hmm, system still not looking right - messages are delaying an hour before delivery. Hrm. Time to fire up the text mode debugger from the runtime_tools package.

(erms@testbox)>  G = group_leader(), dbg:tracer(process, {fun(M,_) -> io:format(G, “~n~p Trace: ~p~n“, [calendar:local_time(), M]), 0 end, 0}).

(erms@testbox)> dbg:tp(erms_smsc,handle_operation, dbg:fun2ms(fun ([{submit_sm, _, Pdu}, _, _]) -> Pdu end)).

(erms@testbox)> dbg:p(whereis(erms_smsc)).

These calls set up a tracer that writes messages like “<time> Trace: <message” to the console (in my case a remote shell, hence the fancy group_leader bits), sets a trace on calls to erms_smsc:handle_operation with a match spec looking for submit_sm arguments, and sets trace flags on the 'erms_smsc' process.

... Lots of trace data ...

... call to submit_sm with the Pdu we want ...

Now I can copy the literal Pdu (the important piece of data that has the delivery and expiry times that are getting confused), and paste it to a local node and examine it closely.

(erms@127.0.0.1)> Pdu = <pasted data>.

(erms@127.0.0.1)> erms_smpp:pdu_timing(dict:from_list(Pdu)).

{undefined,{{2007,6,28},{5,24,28}}}

Well, erms_smpp seems to have the Delivery and Expiry times the right way round. Hrm. I wonder if erms_smsc is swapping them. *check source* Yup, sure is.

*hackety clickety fixety*

scp ebin/erms_smsc.ebin testbox:

(erms@testbox)> code:load_abs(”/home/me/erms_smsc”), code:purge(erms_smsg).

*watch trace data*

(erms@testbox)> rb:rescan(), rb:show().

And now we can confirm it's fixed. Time to shut down the tracing.

(erms@testbox)> dbg:stop_clear().

Now I can sit back, relax, and lament that if my next project isn't in erlang I'm going to have to fall back to stone-age print-f style debugging.