I'm in an Immutable State

(Andy Wootton) #1

I learned to code when ‘algorithms + data structure = programs’. I haven’t ever used OO in anger but I think I understand the basics. My mental model involves the manipulation of state in the form of values, stored in variables.

I’ve used functions and recursion successfully in VAX/VMS Pascal so I understand how functional languages use recursion to replace loops BUT I’m really stumped by immutability. I don’t know how to get some stuff done without variables.

I understand that pure functional languages can’t have side effects, so can’t directly mutate state (but Clojure isn’t pure.) I’ve worked through a few examples that look like a pipe-line of functions transforming one data structure into another but I can’t see how to do some really simple things.

Imagine a shop that sells 1 type of widget and when stock falls below 40% of maximum, re-orders. In an imperative language, I could have a constant for target stock and the percentage and a variable to keep count of how many were left. Assuming a multi-user system, I’d expect to have some sort of transactional locking system on my 1 variable. How on earth do I do it in magic function land? Without parallelism, I can see how a function might recurse and maintain a value within that single function but my function could be called on multiple cores at once. Can anyone help me see what I’m missing?

Now I’ve written all that: is my assumption that multiple calls of the same function are completely independent valid? If the interpreter queues calls to each function, there would be a blocking mechanism but that would be a limit on parallel scalability, so seems unlikely.

(Marc Cooper) #2

You might want to consider agents. See:

(Andy Wootton) #3

Thanks for giving me a search term @auxbus. Agents gave me refs
This is probably the best starting point https://clojure.org/reference/vars

I must admit I’m a bit disappointed that I have to ‘moot’ at all. I think I get it now though. We are reflecting in the realm of magic changes happening in the real world, so there must be mutation.

(Marc Cooper) #4

There doesn’t have to be mutation. I don’t know clojure well enough, but elixir will rebind a variable giving the illusion of mutation (which is what clojure’s binding of vars appears to do). Even so, rebinding is different to agents and their kin. None of these mutate.

(Andy Wootton) #5

You’re right. I’m trying to maintain a single time-sequence of values shared between multiple threads, to reflect a real world state change in the number of items remaining, ‘as though’ a value was being mutated with locking. Thanks.

My first thought was that it would be necessary to sequence the calls to a single function but that prevents parallelism. I guess this blocks slightly too but for the shortest possible time.

(Andy Wootton) #6

My logic is wrong too. Order only matters when there is contention for the last few items.

Because I’ve only been working on toy projects, I haven’t been thinking in terms of shared data structures at all. Rich Hickey often talks in terms of consistent snapshots of data, like ‘knowledge cube’ people do, ignoring that the snapshot is aging. I’ve been uncomfortable about that but haven’t been able to work out why. People taking their own branches of reality is a new version of the spreadsheet problem.

(Andy Wootton) #7

@auxbuss Are you saying that Clojure’s https://clojure.org/reference/agents are different to Elixir’s? That’s slightly worrying. So far, I’ve found the functional languages I’ve seen to be either consistent in concept OR different, not to have overloading of jargon words with subtly different meanings.

[ I’ve read that again - an immutable agent for a mutable location. Maybe it IS the same thing. It sounds the same kind of indirect change as Haskell mono-things ]

(Marc Cooper) #8

In elixir, every process has a mailbox (think message queue), so there’s no blocking. You just join the queue. Agent abstracts that for you. (Actually, Agent abstracts GenServer, which… turtles!)

In case I’ve never mentioned it: elixir is awesome!

(Marc Cooper) #9

They look like the same thing – based on a quick scan of the clojure docs.

(Andy Wootton) #10

Sounds like Hoare’s Communicating Sequential Processes that occam was based on. I think there are other architectures but I’ve never looked at them.
I’ve never read this but I remembered if was Free http://www.usingcsp.com/

I think I’ve been convinced that time is real today. It is the order of events. The old joke “it’s what stops everything happening at once” is a fact, in the special case of enforced sequence, or a ‘value at this point in time’, if you’re a Rich Hickey disciple. We judge the passage of time by observed state-changes.

(Andy Wootton) #11

I’ve talked to a guy 3 times at Web Staffordshire. He used to be a lecturer at Keele. This time he told me wants to develop a language for real-time control and simulation, using the best of Algol 68 and occam. It’s a funny old world.

But back on the immutables: I’ve been looking at Datomic. It extends Rick Hickey’s ideas about extensible value data structures to databases, on top of a host DB. It seems good for systems that are mostly read and in those conditions, linearly scaleable by adding clients. It uses a totally different architecture than anything I’ve ever seen but I suspect it may be related to recent log-based disk sytems that I read about. Nothing is deleted.

Has anyone used it? I’m a bit confused by the pricing model. It seems to be either free with limitations or $6000 ‘per system’. I don’t even know what that means.

(Andy Wootton) #12

Datomic’s definition of an atomic ‘fact’ is a value at a time. The latest value is equivalent to the ‘current value’ of a field in an ‘imperative database’ but all the previous versions are held too, allowing time-travel. Great so far but:

A datom is “an immutable, point-in-time fact: [entity, attribute, value, transaction (TIME really), added (or RETRACTED)]”

I’ve never felt it is possible to distinguish cleanly, for all future cases, between an entity and an attribute.
e.g. I’ve been looking at vertex/edge graphs from discreet mathematics. What are the entities, the vertices, the edges or each end of the edges? Obviously I’m confusing the model I choose with the object I’m modelling but isn’t that the point of an object model? If I can’t do that, I’m storing data about my model not my observations of reality against a simulation of reality. I’ve been thinking of graphs in graphical form, how I normally deal with them but they aren’t necessarily real. They are a mathematical concept of connected nothings. Magic again.

Exhibit 2: Point A, Point B and the distance between them. What is that distance an attribute of? It’s a thing we invent isn’t it, which is subject to change in the state of either? Or it’s a computation, whenever needed, from the position of the 2 objects so was never an attribute in the first place.

I think I’ve just argued myself into a corner where nothing in software is real, again, so none of this matters:
An entity is a thing, possibly imaginary and the attribute is a name for some value about it we want to store. I think my subconscious concern was that we were associating a value with one thing when it might be shared between several but this is Clojure, where one thing can be a collection of several things. I’ll be OK now, I think.

[‘The Ultimate Question’, ‘The Answer’, 42, ‘Earth time: 1978’’]
is a ‘fact’, expressed as a datom. What meaning we choose to project onto that is entirely up to each of us. With reference to some shared culture, we hope everyone will choose the same meaning.

(Dee Vious) #13

Have you found out that Datomic is free, yet?
For all personal use, it’s free. The limitations on its use have been revised and enable you to host a datomic server app, or two.

Unless you are building an IT empire around Datomic, it’s free. And, if you are building that empire, you should be in a position to pay for it’s global use.

I am looking at using Datomic, when I get around to learning Clojure. Previously, I’d been considering ArangoDB but, the code-is-data philosophy, employed by Clojure and Datomic, appeals to me, greatly.

Similarly, I had been considering Apache Spark for processing data. Now, I’m all about Onyx Platform. Onyx describes itself around your data, using a ‘catalogue’. This is analagous to Datomic and its use of a schema.

(Andy Wootton) #14

I had got that impression, yes. I’m not sure if that was always the intention and it wasn’t explained very well. I agree that we need to find a model which pays FOSS authors when people start to make a profit from their work but it stays free for people who don’t.

(Dee Vious) #15

It always had limitations, which put me off but, they’ve been greatly relaxed. To the point where I can envision Datomic in my global empire!

Plus, Datomic works very well with Datascript, on the client. These tools have a lot of potential but, so do I! I need to convert some of it into kinetic energy, and start coding.

(Dee Vious) #16

The Datomic Free transactor is limited to 2 simultaneous peers and embedded storage and does not support Datomic Clients. There are two ways to consume Datomic: with an in-process peer library, or with a client library.

Datomic Starter has unlimited peers and clients. New users might begin with a client library - Clients are a better operational fit for microservices, where application processes may be small, numerous, and short lived.

Peer processes should be large, relative to the size of the (possibly virtual) hardware they run on. This amortizes the cost of running the JVM, and allows for large object caches. Peer processes must be JVM-based, and include several Java library dependencies.

Clients have a much smaller footprint - I’ve yet to figure out the implications of ‘embedded storage’.

Onyx data processing uses 'tagged’ peers, enabling it to stay within Datomic’s free licence.

The Datomic Starter license includes high availability, memcached integration, and Datomic Console. The licence is perpetual, but limited to 1 year of maintenance and updates.
So, after a year, you have to get off the pot, so’s to speak!