It's November, and we have a lot to be thankful for. It's been a grand 15 years since we added the 2 in Everything2. It's been so long that the original Everything is but a glint in history's eye. We paved the way for a lot of modern websites, and it's been an incredible 15 years. Safe to say that this website hasn't exactly kept up with the times in terms of functionality or looks, but we still love it. This log is going to talk a bit about how we're modernizing things from here, and what this all means.
Last month, in root log: October 2014, I said that I was going to be working on writeup feedback, and I still am. I am taking a deep dive into the display code for how the writeup pages are generated. This is advancing a bit more of the modernization work I am pursuing, and it's all in all, a very good thing. I'm hoping to deliver on it this month, after I unsnarl some of the stuff in show content.
I haven't been root logging as much, but work is still continuing. A lot of work is happening in the lab, but I'm going to try to be more up front about what's changing. We've got 15 years of cruft to undo, but I'm making steady pace on it day by day. The major push right now is to refactor the display items into controllers. This is going to use the basis of the emulation work that I've put forward before. All of the moving things into Everything::Application and the work pushing htmlcodes and opcodes into Everything::Delegation are going to help immensely in how this goes down. Some of the layers can't be fixed until after we have a full controller stack working for us. For instance, maintenances are too entangled into the Nodebase objects to pull out until after we have those update handlers nailed. My original method for doing this was to make it so that we had strong objects (the M in the MVC paradigm), and we could refactor a lot of the code into that. It is pretty clear from here however that we need to separate display and controller logic first.
This is going to accelerate the Mason (templating system) work, and the more extensive push to use Moose is going to come after that.
Time to shave some yaks. I'll start adding updates from here on down
Nov 1-15:
My day job was a bit crazy in here, but the gist is that I experimented with various Controller schemes and dug into how writeup feedback could be added to the codebase without causing an immediate run-up in technical debt. Instead, I refactored everything that could leave Everything::HTML over to Everything::Application, and that leaves only the page routing and generation code inside of it. This leaves us with a good point to put a fork in to move things into a template scheme, nodetype by nodetype. The rest of the time I have spent assembling the starter template divisions from scratch. This means moving a lot of the code that is in nodelets and htmlcodes over to a controller/sub-controller setup. We can emulate execution if we need to, but generating pages without using eval() is the end goal of this exercise.
Nov 17:
Finished removing the legacy setting of 'settings_useEpicenterZen', which is now on by default. 72 accounts weren't converted, so I did them by hand, and removed all mention of the setting.
Removed the settings script section from Nodelet settings. Deleted settings script
Nov 18:
Deleted the nodecache nodelet. There are other, more secure ways to look at the nodecache in production. Cache dump being the real way to do it. Having something that would / could run every pageload doesn't make any sense.
Nov 19:
Template work in development now includes the entirety of the zen container chain. We can fully render the page out of Mason properly, including a lot of display variants. We do not properly include some of the mobile stuff, as that is like instantly going to be thrown out.
Moved browser logging from the Epicenter (no kidding) over to Everything::HTML, where it will eventually be put into the request class, after we don't need to use the display fork in development.
Simplified zen stdcontainer in a few places by using the $APP functions instead of inline stuff.
Nov 20:
Epicenter now works in the template environment (which is off by default).
Nov 21:
Starting to do another round of cleanup on the configuration system. Need to de-nest it to make it it's own Moose object. For that reason, starting to move stuff out of $CONF->{system} to just $CONF. The guest_user piece is done. It's a bit of a snarl since it's literally a 6 step process to move it over.
Nov 28:
All of the CONF hashref stuff is out of the codebase, replaced with a really convenient Moose object. Now working on getting some nodelet stats, so I can work to thin out the ones that aren't going to be supported anymore. We need to push them into templates/controllers now so that I can start to get the site to be moved over.
I've done a survey of how many of each nodelet are in active use by every profile on the site:
Epicenter (262): 118173
New Writeups (263): 117450
Vitals (165437): 117118
Chatterbox (170070): 117040
Other Users (91): 116180
Personal Links (174581): 57228
Random Nodes (457857): 56210
ReadThis (1157024): 31199
Messages (2044453): 4262
Statistics (838296): 4246
Everything Developer (836984): 1179
Recent Nodes (1322699): 1165
Notelet (1290534): 1028
Current User Poll (1689202): 918
Quick Reference (1685943): 487
Notifications (1930708): 288
New Logs (1923735): 166
Categories (1935779): 149
Cool Staff Picks (1922511): 144
Favorite Noders (1876005): 138
Usergroup Writeups (1924754): 118
New Writeups - Zen (1868940): 70
Master Control (1687135): 34
Most Wanted (1986723): 28
For Review (2068913): 16
Node Statistics (90): 7
Server Statistics (93): 7
Tick tock (94): 7
%$NODE (85): 5
nodelet (170): 1
Sign in (2029388): 1
Recommended Reading (2027508): 1
Social Bookmarking (1926311): 1
CGI::param (87): 1
It's pretty clear that anything under 100 users needs to go, with the exception of the editor nodelets. That's very helpful in trying to figure out what's being used.
I've nuked CGI::param, since that's been replaced by literally every web developer browser plugin made in the last decade
Taking a look at the nodelets which are updated by the cron job:
mysql> select nodelet.nodelet_id,node.title,nodelet.updateinterval from nodelet left join node on nodelet.nodelet_id=node.node_id where nodelet.updateinterval > 0;
+------------+---------------------------+----------------+
| nodelet_id | title | updateinterval |
+------------+---------------------------+----------------+
| 93 | Server Statistics | 5 |
| 94 | Tick tock | 60 |
| 447718 | cool nodelet | 200 |
| 618679 | reallycool nodelet | 300 |
| 632556 | tfcool nodelet | 30 |
| 1124969 | tfcool nodelet2 | 200 |
| 1216418 | New Writeups Feeder | 60 |
| 1217114 | New Writeups Normal | 120 |
| 1252384 | Random Nodes XML Feed | 30 |
| 1801407 | Test NW Feeder | 300 |
| 1822002 | New Writeups Atom Source | 120 |
| 1876900 | Cool User Picks | 30 |
| 1882821 | Cream of the Cool | 300 |
| 1916666 | Staff Picks | 30 |
| 1927182 | Logs | 500 |
| 1995929 | New Writeups (guest user) | 300 |
| 2002281 | Front Page News | 300 |
| 2027488 | New Cool Stuff | 600 |
| 2027508 | Recommended Reading | 1200 |
| 2051342 | Neglected Drafts | 86400 |
| 2068913 | For Review | 360 |
+------------+---------------------------+----------------+
Nuked Tick Tock, as only 7 users were using it, and it is one of the old ecore nodelets
Nuked Server Statistics, as only 7 users were using it, and it wasn't actually representative of anything. It would get updated from the bastion host where all of the cron jobs run, so it'd always show artificially high load (as it would get evaluated in the middle of a cron job batch).
Nuked %$NODE because it was only used by a few retired coder gods, and I don't think we need to port it.
Nov 29:
Working through the codebase to wind down all the areas in which New Writeups - Zen existed. A transition layer was enacted inside of nodelet meta-container, but it hasn't gotten to everybody, so time to do some data manicuring.
Manicured everybody's settings to replace New Writeups - zen with New Writeups
Nuked New Writeups - Zen
Did some initial investigation work into data stash stuff. This is going to replace the way that the HTML gets stashed into the database inside of the nodelet code. I've long thought that the JSON parser (implemented in C) is probably considerably faster than my more performant, but still lacking Everything vars parser, which is stored as a CGI string. Running a quick benchmark, it's pretty damn obvious:
Benchmark: timing 100000 iterations of Everything vars, JSON...
Everything vars: 12 wallclock secs (11.47 usr + 0.01 sys = 11.48 CPU) @ 8710.80/s (n=100000)
JSON: 2 wallclock secs ( 2.54 usr + 0.00 sys = 2.54 CPU) @ 39370.08/s (n=100000)
~5 times faster.
Nov 30:
Put in place the start of the datastash code, which is a straight up replacement of nodelets, and eventually settings. newwriteups is the first such stash, now powered by a new cronjob which will execute all of the Everything::DataStash objects it finds in @INC. This will allow me to star to clean up these old jobs.
Deleted New Writeups Normal, an old backup I made when I put together one of the current New Writeups code pieces.
Nuked the cool nodelet, as it isn't used anywhere, as far as I can tell, and no profile has it
Nuked the New Frontpage, which was a beta for what eventually became, as far as I can tell Welcome to Everything
Nuked Welcome to Everything2.0, which was another beta cut at a front page
Nuked reallycool nodelet, which was an old layout style and was only used in earlier incarnations of the frontpage