Scrolls: Unraveled

Documents grow, shrink, and mutate in ways far more engaging than what was likely ever conceived for early web-browsing. In modern spirit, we need not be afraid of getting a bit exotic with the way we present classic constructs in our web-pages.

YouTube Live Chat Stream

So long as we use current web tools, manipulating DOM nodes to present content for users, we are responsible for making the experience as seamless as hardware allows. As for content, let's take the humble list; a collection of items with content, calls to action, and expected gestures belonging to each item is easy enough to create, show, manipulate and so on.

We throw as many elements at our document as we need. That's fine, plenty of pages on the web append elements indefinitely. Realistically, most users never actually stick around long enough for thousands of items to update, or those thousands of items never do much in terms of user-interaction. In most of these cases, the requirement is minimal, so we don't usually have to worry about performance.

We rely on the metal in to which we throw our programs to churn through however many steps be required, only to care for the final result. That's fine, it's exactly what pen and paper does for a busy mind, and exactly what a computer is ought to be doing whilst it shepherds our data on the digital frontier. But, as we get ever-closer to squeezing the last drop from silicon, we ought not rely on sheer force, or scale, to solve problems that can otherwise be framed alternatively—achieving the same, if not better, result along the way.

Growing Pains

There is quite a bit of literature on the topic of scrolling, content absorption, action-reward mechanisms. Most of it attacks, quite rightly, the psychological and physiological effects of ever-enticing scrolling experiences. For those interested in the effectiveness of such sorcery, plenty can be found on—yes, the Internet! I'd like to avoid fixing that problem in this exercise. Although it's a valiant fight for keeping one's mind at peace, it's oft a fight each must lead with a firm will in this noisy world.

That said, fear not, as we are not writing refined programs, only to find they will enslave ourselves with their might! For now, we can embrace the computers we are teaching, and firmly believe their rigid abiding nature will alleviate more inconvenience than they create trouble.

In the Now!

With each historical step towards presenting content, the delivery-method must be made equally pleasant! Live radio and television have been refined to wonderful levels of engagement for the audience they seek to inform; in similar fashion, our web application must make this effort anywhere we can justify its efficacy.

Where the visual/auditory sensation of content-consumption seems most-rewarding, we should be thinking about making that delivery as fluid and natural as possible. Easing looks better than abrupt stops in animation because that's how most things work in real-life. This topic comes up a lot in physical mechanics, architecture, where actual things behave according to natural laws. More on the Everything is a Spring topic can be found in plenty of articles on the Internet.Disclaimer: a little bit of math and physics follow down that rabbit hole, but it's a really interesting topic.

The reason any of the math or physics involved in describing physical systems, is that we're replicating one in our browser. The window, viewport, or whatever we want to call it, displays list items. If we are to engage with this list in any way, it should behave the way, say, some wood blocks (messages) on an actual desktop (viewport) would move about the desktop's surface.

If we begin to display our messages with hesitation, the illusion of a physical system is lost, and we reveal the jagged edges of our simulation. It may seem trivial, a miniscule metric that even the user may fail to articulate meaningfully, but it does accumulate—the sour feeling from delayed, jagged, or slow transitions is noticed. The way something feels goes quite a bit further for a useful system than just aesthetics. This, again, is worth thinking about, just not in this scope.

Simply put, if our hardware cannot smoothly show an endless list of elements efficiently and consistently, we can attempt to re-frame the problem—hopefully without the never-ending list of DOM elements.

A Long List need not actually be Long

Let's look at what makes and characterizes a list on a webpage:

One or more elements with meaningful content inside
Organized or ordered by one or more rules (time, alphabet, popularity, importance)
There are only so many elements we can display in a viewport to say they're relevant, legible, or otherwise meaningful in some way to the reader.
Scrolling up and down a list feels natural; a element's position on screen, relative to others, is generally fixed and unchanging—which is to say that we care about an element's persistence throughout the document.
The browser window, or container of our elements usually takes care of hiding/showing elements—we need only supply elements.

We can infer from the last point, that we never need what we don't see in the viewport, which means we can ignore those nodes entirely, as we scroll down the page. Ignoring nodes, usually means relying on the viewport to obscure them from the current scroll position. This is brilliantly simple, but it's a mechanism we take for-granted.

When we consider performance for a long list, the amount of nodes in the document becomes the apparent bottleneck. Sure, the browser obscures those nodes just fine, but they are still written to memory, they are still there, and they will affect rendering of new ones when there are enough existing elements in the DOM. So, if we reduce the amount of nodes, and not rely on the viewport to hide/show elements, the burden shifts to the program.

There is a cost to playing the magician if we are to not rely on the viewport to show/hide elements when we need them for any given scroll position. The more we explore alternative ways of maintaining a long-document illusion, the more we must think about nuanced positioning for which we are now responsible.

Perhaps that effort exceeds the savings in drawing hundreds of DOM nodes in a message stream. However, this exercise seeks to stretch the imagination a bit, and—perhaps it can lead to other interesting solutions for experiences with the ever-growing lists we now consume.

DOM Elements Dance

If we say a message must contain an image (or a tombstone avatar,) date, user name, and some padding/margin, we can assume each node will be some known height in pixels.

For any modern screen, even in portrait mode we care only to the extent of a fullscreen run at a practical limit of 3440px.

Luckily, most mobile devices fall far short of this upper-bound. So, even at that resolution and scaling, 3440 / 150 gives us ~20 nodes. Taking in to account margins between each node, we need even less nodes to fill the viewport.

We'll be extra safe, and, for sake of implementation bump that to 30, which gives us three even groups of ten and makes for some easier transforms later.

We now have thirty nodes, we've scrolled to the bottom of the doc, what do we do? Add more! Well, sort of.

We need a way to scroll down our page, and maintain an actual scroll position as if we had kept all of our messages we've scrolled past. If we don't somehow replace the space lost, we are simply swapping nodes and we never give ourselves a document that grows with each scroll.

As luck would have it, we do know the height of our swapped messages. Since we're iterating through those swapped blocks of messages, we can also keep track of the overall height lost, and figure out a simple way to keep our document afloat as we scroll further down.

Sentinels!

Scrolling Down

A watchful, obedient program makes for a fine element at the top of our document. As we replace nodes, its job is to keep track of the overall height of the replaced nodes and occupy any voids created by removed nodes.

Since the height of any one message can vary—influenced only by the length of the message's content, we sum the height of all messages in the swap operation, and extend our sentinel's height by that amount.

Each time we swap blocks of nodes, whitespace is preserved, and the document grows naturally—as if it had all previous nodes where they were initially. Our scroll position, from the viewports perspective evolves as we would expect—maintaining the illusion of a long list of DOM elements.

Surprisingly, that should cover basic scrolling through long lists of messages. With relatively little effort we can shuffle a handful of elements, quite quickly, and with no sight of edges of the smoke and mirrors involved.

Optimizations and Further Implications

The aforementioned cost presents itself to be paid in this section. We're not just scrolling through a long document of infinitely-added elements. We divide a fixed set of elements, introduce states of transition, and rely on correct arithmetic to choreograph our messages to dance in unison. JavaScript, waving its best flag, gives us flexibility and smoothness in its async execution, but that manifests itself in quite the unexpected behaviour.

Dynamic Heights

The problem evolves when we consider a fluid viewport. Were the viewport to change in width, the available width for message content pushes the messages taller—increasing overall swap heights. Depending where we are in this list, the height we we were tracking will not correspond to an updated height-per-message in the re-sized viewport. So, when we scroll back up, our sentinel height calculations fall out of sync.

We can guarantee a final zero state for the sentinel's height when we do scroll all the way back, but along the way we may encounter whitespace from the longer-than-expected sentinel.

Suppose we start with a smaller viewport and scroll our document a few iterations. We resize to a wider viewport and scroll back up. The previously accumulated sentinel height will now be larger per-message since the smaller viewport was skinnier—pushing message content, and by extension, the entire message height. So, when we operate on the new, wider and shorter state, we subtract less than we should from our sentinel.

Impatient Events Queued into Submission

One of the now popular gestures associated with list in web applications is a swipe. This is not new to our daily routines with paper documents, dust on our tables, or bugs in our face. If we do not like something, we swipe it away, out of sight and out of mind. So, when we introduce swiping to our list, we almost certainly think of dismissing a particular message.

The thought is relatively short-lived in our minds, and our expectation is simply for our wish to be fulfilled. To our program, however, our wish can only be fulfilled by the exact conditions we specify.

Simple conditionals dictate what we should do with a message when it is swiped. A swipe event meets a minimum threshold, the message moves off-screen, done. Well, not quite. We want to keep the actual DOM element we've removed from view, and replace it with something more meaningful for later viewing. That's fine. We can move the message to end of the list and queue up our next message, from the network, or cache. Still nothing really novel.

But, suppose we are impatient and swipe several messages from view and quickly scroll up or down in the document.

In the time, or state, where our messages are being removed from the list, we scroll down, and we tell our program to swap a block of messages from the top, to the bottom. The program must interpret the top most set as 'ready for swap,' yet we could have a swiped message in that set. So, before we've removed and replaced the swiped message, we swap it to the bottom-most position from a competing scroll operation.

This breaks continuity in most cases, as whichever operation completes first, the following one operates on an outdated state and populates the replaced node with incorrect data.

The precedence of swipe over scroll can likely be proved trivial, but in practice, the scroll queue never appears to reach more than one operation—likely because we can't scroll that fast, but more importantly, because we shouldn't be able to scroll past an unfulfilled scroll-swap operation to even need another. That would imply that we've reached past the visible nodes of our growing/shrinking document, and would undo what we're trying to build.

A technical implementation of this queue can be reduced to nothing more than a RequestAnimationFrame loop operating within the scope a few states:

Queue Instance

Property	Value	Usage
this.pending	`Array`	A collection of pending requests to be executed in the order they were added.
this.isRunning	`Boolean`	A general state for the queue instance, indicating whether all pending requests have been fulfilled

Each pending request is then constructed of the following:

Request Object

Property	Value	Usage
target	`Object`	The relevant DOM element upon which we are to operate.
hasFinished	`Boolean`	A general state for this particular request. Evaluated when previous requests have been fulfilled. Satifisfied when the passed condition is met, or instantaneously if no condition is passed.
active	`Boolean`	A general state for the queued operation. Used internally to loop RAF until the condition is met.
run	`Function`	Code to be run once our request has been fulfilled.

Subsequently, a function operates on the accumulated requests, in-order, until they are all fulfilled, and the requests array is emptied. So long as our run function takes less than the 16ms it should take for a RequestAnimationFrame to complete before the next RAF is called, we should be able to maintain strong synchronization among our draws, swaps, scrolls, and swipes.

So, with a queue instantiated for each distinct action, scroll and swipe, if we have a running swipe queue, we hold off on replacing DOM nodes for a scroll event that may be requested. We fulfill the scroll request after the swipe operation has settled our recently swapped/replaced DOM element, and allow it to operate on the most current state. This appears to happen fast enough for a practical amount of swiped messages. It can likely be broken, but likely not at the speed with which normal fingers can operate.

Network Optimizations:

Since our presentation of messages, under-the-hood, is more akin to pagination, we can leverage our dynamic calls for new messages in at least these fashions:

We're already looping through our messages, so we can push to an array with message contents and any other meaningful attributes about it—replacing said info from the cached array on subsequent swaps. So, in the same operation, we simply check to see if the message is already available, and populate data from the cache, rather than sending off another XHR.
We can extend the reach and power of our cache to utilise robust search by relying on cached data, rather than server-side look-ups. So, where a robust lookup may take longer to fulfill, a similar algorithm can be executed clientside with the already-available data. Whether this is actually true becomes evident in future iterations of a tool like our dynamic list.

The above satisfy server load concerns, but what about user experiences in slow-internet conditions. The state of mobile Internet in the very country that invented it can be—slow. At least for our scope, a set of messages is usually a very small amount of data. However, user avatars, gifs, images, and other engaging content can amount to more expense.

Tombstones over Spinners!

Furthermore, the whizzing motion of the tombstone demonstrates motion in general—exactly what a spinner is meant to accomplish. Where the latter is simply a visual cue that some request is being fulfilled, the former is interactive and can delegate multiple requests, with more granularity to changes in direction, speed, and precision.

This construct not only maintains continuity in the visual sensation we get from scrolling, but also allows for a more robust search when those messages finally resolve to data. Where the former allows the user to actually associate their actions with instantaneous feedback of a scrolling list of elements, the latter opens up possibilities in search that are often ill-represented in search/filter applications.

When we search our list, we usually recall a few words, and hope to find those words in a message we or someone else mentioned in the last couple days. So instead of waiting for a single set to be retrieved, we retrieve several sets with a continuous gesture, and then go jazz-hands on the search bar. In fact, this style of fetch a bunch—search when done dynamic can allow us to quickly realize what it is we want from the list. The eventual search operation can then programmatically scroll us right over to the message(s) we are hoping to find.

These are simply ideas for the direction a project like this would go; After some thought, implementing tombstones by resolving a Promise bound to some DOM element is likely just the simple case. Were we to scroll past a few swap operations, the node to which we would resolve our data would already have been swapped and may be in a different place entirely. We would need more logic to dismiss the promised data and populate from the relevant request. That is not to say this is impossible!

All Together Now!

Below you'll find the full implementation of this little dance of DOM elements. Be sure to try and break it, scrolling as fast as possible and swiping to dismiss messages all at once! Pulling up dev tools will reveal how things are being swapped. Source code, and suggestions are welcome on GitHub can be found here.

I am confident that there is efficiency to be brought to long-document scrolling experiences. This may be exotic, or not exotic enough, to actually leverage the best possible performance and usability, but I do believe it helps to re-frame classic constructs in our documents.

Have fun and be sure to suggest improvements, question the efficacy of all this swap-nonsense, and anything else that comes to mind!