Storing program source as relations in a database instead of text file

[PeterDonis]

Otherwise, supposedly, it would get too complicated to represent as relations.

In other words, what you originally claimed in the OP and in the title of this thread is actually impossible?

 

[elcaro]

In other words, what you originally claimed in the OP and in the title of this thread is actually impossible?

It would not be impossible in theory I guess. But it would imply your database is able to store the exact details of the AST of the grammer of the programming language (and specific version) you use. The issue is not so much if that is possible to implement, but what the benefit of that would be in terms of costs, etc.

 

[PeterDonis]

it would imply your database is able to store the exact details of the AST of the grammer of the programming language (and specific version) you use.

You would be storing them as relations in the database. That's what your OP and thread title said. Every database can store relations. Are you now doubting that the exact details of an AST could be stored as relations in a database?

 

[elcaro]

You would be storing them as relations in the database. That's what your OP and thread title said. Every database can store relations. Are you now doubting that the exact details of an AST could be stored as relations in a database?

Didn't say that. Theoretically it is doable, but quite complicated. The only issue is if that is worth doing and needed for the purpose of the tool.

 

[elcaro]

You would be storing them as relations in the database. That's what your OP and thread title said. Every database can store relations. Are you now doubting that the exact details of an AST could be stored as relations in a database?

It is a similar problem as let's say storing adress information in a database. You could store it in specific tables, a table for country, city, street, postal code, etc., but that requires you have a complete table of all countries, all cities and all streets and postal codes that exist around the planet (and update them every time this info changes), for storing your adress information as relations. A more simple storage format is using text fields to represent the adress.

Would you then also argue that unless the adress info is also stored in relational form, there is no use in using a database for storing customer information?

 

[PeterDonis]

Theoretically it is doable, but quite complicated. The only issue is if that is worth doing and needed for the purpose of the tool.

Since the stated purpose of the tool is to store program source as relations in a database, it would seem to me that if anything is not stored as relations, the purpose of the tool is not being met.

It is a similar problem as let's say storing adress information in a database.

You didn't say "store program source in a database". You said "store program source as relations in a database". Big difference.

 

[elcaro]

Since the stated purpose of the tool is to store program source as relations in a database, it would seem to me that if anything is not stored as relations, the purpose of the tool is not being met.You didn't say "store program source in a database". You said "store program source as relations in a database". Big difference.

Yes, but nether storing programs as text nor storing it as relations in a database is a goal in itself, the real goal is of course in terms of being able to develop and maintain programs in a better way, increasing productivity and lowering costs, etc.

 

[PeterDonis]

nether storing programs as text nor storing it as relations in a database is a goal in itself, the real goal is of course in terms of being able to develop and maintain programs in a better way, increasing productivity and lowering costs, etc.

Then what is the point of this thread? Are we discussing storing program source as relations in a database, as the thread title says? Or are we just waving our hands about stuff that might or might not improve various things about programming?

 

[elcaro]

Then what is the point of this thread? Are we discussing storing program source as relations in a database, as the thread title says? Or are we just waving our hands about stuff that might or might not improve various things about programming?

Database are used by definition for storing relations. If the only thing we did was using the database for storing text as text, we would not need a database.
But there is not perse a requirement that ALL relations are stored in a database (see the example I gave in a previous post about storing adress information in a customer system) for the use of a database to still be usefull.

The only thing we can argue about is wether or not that that is the case in this particular case for storing program source.

I gave some arguments why it still could be usefull (for generating automatic build/make scripts for example) for that purpose, without the requirement of breaking the program source down to relations at the deepest level.

 

[FactChecker]

It sounds to me like you are talking about using a database to hold the information that is in a "set/use" table. There are problems with using that to represent the entire program code. Here are a couple of thoughts that I have on the subject:
1) As others have said, it is hard to imagine any other way than code to represent the details of what a function actually does. Something like "x=y+z" is already simply represented in the code.
2) A library is a database in that a calling program can look up functions it needs by name and execute it. There is a lot of work that has been done along those lines (including OOD) to allow the correct functions to be retrieved from the correct library.

 

[valenumr]

I think the only fundamental issue to answering this question is whether modern sql is Turing complete. I'll go with "maybe", but I think it would be painful and terribly inefficient to try and create a complex program with this approach.

 

[Klystron]

The OP mentions (and dismisses?) smalltalk computer language but also consider LISp and offshoots such as CLOS. (I mentally substituted lists for relations while attempting to understand the original question.) C and its derivatives and FORTRAN were designed to be sparse, particularly considering hardware limitations, then and now.

Very large software systems are commonly databased including templates for coding applications, platform dependent system code, test cases, and documentation.

 

[phinds]

Obviously you've never programmed in C and had to write a header file. I wish I could say the same, as it would mean I would have avoided much pain and frustration.

GADS. It's been so long since I've done C/C++ that I forgot all about it.

 

[phinds]

This is common C and C++ parlance. It's basically a declaration of the function, including the number and types of parameters, and the return value type.

See post #48 directly above

 

[valenumr]

GADS. It's been so long since I've done C/C++ that I forgot all about it.

C++ is evolving rapidly on good ways.

 

[PeterDonis]

Database are used by definition for storing relations.

They can store relations, but that is not the only thing they can store. The term "relational database" does not mean that the only thing the database can store is relations. It just means the database engine is optimized for dealing with relations.

The only thing we can argue about is wether or not that that is the case in this particular case for storing program source.

I'm still confused about the point of this thread. By your OP and the title of the thread, I had understood you to be proposing that program source be stored as relations in a database. Is that what you're proposing, or not? Does the title of the thread need to be changed?

 

[valenumr]

They can store relations, but that is not the only thing they can store. The term "relational database" does not mean that the only thing the database can store is relations. It just means the database engine is optimized for dealing with relations.I'm still confused about the point of this thread. By your OP and the title of the thread, I had understood you to be proposing that program source be stored as relations in a database. Is that what you're proposing, or not? Does the title of the thread need to be changed?

I was under the impression that the question is if a dbms can execute an arbitrary program. Otherwise, we have things like uml to model a lot of the concepts being referenced, which certainly can be stored in a database.

 

[PeterDonis]

I was under the impression that the question is if a dbms can execute an arbitrary program.

I don't know where you're getting that impression, since it is not what either the thread title or the thread OP says.

 

[valenumr]

I don't know where you're getting that impression, since it is not what either the thread title or the thread OP says.

Maybe a leap of logic. If the system can fully describe a program, it isn't a huge stretch to think that it can interpret it, but perhaps that's a stretch too far.

 

[Mark44]

I gave some arguments why it still could be usefull (for generating automatic build/make scripts for example) for that purpose, without the requirement of breaking the program source down to relations at the deepest level.

But the thread title is "Storing program source as relations in a database instead of text file." Since you seem to have given up on that idea, it would probably be best to give this thread a different title.

 

[valenumr]

Here is where it gets a little weird to me. If one can store a program description, I think it follows that one should also be able to retrieve the program and all the information it contains. And if one can extract all information of the program as stored, one should be able to interpret it's intent.

 

[FactChecker]

Here is where it gets a little weird to me. If one can store a program description, I think it follows that one should also be able to retrieve the program and all the information it contains. And if one can extract all information of the program as stored, one should be able to interpret it's intent.

IMO, it is certainly impractical to store all the information in a relational database. I won't say it is impossible, but it is definitely beyond my ability to envision.

 

[Tom.G]

Rhetorical Question:
How does the initial proposal vary from a Flow Chart?

 

[synch]

...
But intrinsically a program can also be seen as a collection of relations between different objects, which can be stored as tuples in a relational database system.
...

The relations between parts of the programs structure no doubt could be stored as tuples in a relational database using SQL. If I remember correctly, the relational model insists that the mechanisms for database maintenance are also implemented and stored relationally so I guess the usual RDBMS is a sort of example already. BUT... most other tasks and programs have to use extensive sequential and procedural logic, which is not a database strength, in fact, even storing derived detail is frowned on by RDBMS analysis unless it is a speed optimisation of some sort.
I can see the logic in the idea, but practically it would be a nightmare. Programs would have to be written in SQL. Eg to add two variables - the SQL program would have to apply read locks, access the variables' current value in tables and use the + operator in the call, yada yada yada ... the SQL would be horrendous after a while and it would run very slowly, with most RDBMS not exactly optimised for non-standard use. And that is avoiding the question of defining operators' actions in what contexts and so on.

However the logic certainly has value - I have found Codd's excellent rules for RDBMS are also surprisingly good at promoting good program structure, when (loosely) applied to writing program code not anywhere near RDBMS :) !
Eg thinking of a line of code like a row of fields in a table, and transactions (as far as realistic) - a line should not have dependency within the line, a line of code should achieve one action only, sequential lines of a composite action should be in a defined block that can be tried as a single action or rolled back, and so on . (Not surprising when you think about it - just IMHO)

 

[jack action]

the SQL program would have to apply read locks, access the variables' current value in tables and use the + operator in the call, yada yada yada ...

I must admit that I was interested in the OP idea and I had problems following this discussion as I wasn't sure I understood other people's criticisms. But I never got from the OP that the program would be run by SQL. Just like no program is run directly from a text file.

What I understood was that instead of saving a text file, you saved everything as objects stored in tables - with names like 'variables', 'operator', 'function', 'control_structure', 'function', 'classes', 'namespace', 'expression' - all of them having some attributes or requirements. Of course, it will still have to be compiled before being executed. The compiler will have to already understand that, for example, using the operator with operator_ID = 1 means that it must perform an addition (or whatever). That is no different than a compiler reading a text file and it understands that when reading a '+' means that it must perform an addition.

The interesting advantages cited in the OP are that you can:

• follow the relations and enforce the rules before compiling
• automate certain processes (ex.: if function A is used, called library X)
• study relations in complex programs more easily
• let programmers choose their own language (or maybe even their own grammar; you could choose that an expression is ended by a semicolon or a new line) when converting from/to a human-readable text file (maybe even reading/writing flow charts)
• etc... (as I'm basically restating the OP)

It doesn't affect the speed of programming as, in the end, the SQL file must still be compiled to a binary file to be executed.

I guess the basic concept is that it would be storing the code in a state just before code optimization is done in a compiler, maybe even just before machine code generation. So preprocessing, lexical analysis, parsing, and semantic analysis are already done prior to saving the file. This would reduce file size and also reduce compilation time. Think of Javascript programs that are compiled (and maybe even downloaded) every time a web page is open.

 

[FactChecker]

The interesting advantages cited in the OP are that you can:

• follow the relations and enforce the rules before compiling
• automate certain processes (ex.: if function A is used, called library X)
• study relations in complex programs more easily
• let programmers choose their own language (or maybe even their own grammar; you could choose that an expression is ended by a semicolon or a new line) when converting from/to a human-readable text file (maybe even reading/writing flow charts)
• etc... (as I'm basically restating the OP)

It doesn't affect the speed of programming as, in the end, the SQL file must still be compiled to a binary file to be executed.

My main complaint is that it seems to suppress the order of execution, which I feel is usually the primary information in the code text. And if the sequence of execution is the main "sort order" in a database, I don't know if I would still call it a relational database.
Of course, it is still possible to build relational databases that are used as part of the compilation process. For all I know, that may already be done.

 

[jack action]

it seems to suppress the order of execution, which I feel is usually the primary information in the code text. And if the sequence of execution is the main "sort order" in a database, I don't know if I would still call it a relational database.

The order of execution wouldn't be different. It is about how the information is stored (which includes the order of execution).

Imagine each library stored as a database. Looking for information through the different libraries is much easier and can be done in a million different ways. You can find easily in a project if a library (database) is not used anymore. You may easily search a library that you already have for something you want to execute, instead of loading (or creating) a new library. You don't look for text (which could be written in a million different ways), you look for relations.

The 'relational' lies in the storage, not in the execution.

Of course, it is still possible to build relational databases that are used as part of the compilation process. For all I know, that may already be done.

What the OP says is: «Why not save the programs as those relational databases instead of the original text files? Much likely the file sizes and the compilation time will be smaller and the access to information will be more flexible.»

 

[Mark44]

Imagine each library stored as a database.

This is already done with compiled languages like C and C++. The database has the form of a lookup table whose keys are the function names (C) or function signatures (C++). Associated with each function name/function signature is the compiled machine code for the function.

Looking for information through the different libraries is much easier and can be done in a million different ways. You can find easily in a project if a library (database) is not used anymore. You may easily search a library that you already have for something you want to execute, instead of loading (or creating) a new library.

There already are tools and techniques that can be used to determine which functions a library exports, so I don't see how the suggested changes do anything that isn't already done.

You don't look for text (which could be written in a million different ways), you look for relations.

For the compiled languages I'm talking about, the library (either a static library or dynamic link library) doesn't contain text -- it contains just the machine code associated with the functions plus any other objects that the library exports.

 

[FactChecker]

The order of execution wouldn't be different. It is about how the information is stored (which includes the order of execution).

So, would the order of execution be stored as a number in some of the entries, or as a pointer to the next line to execute, or what? IMO, it would not be a relational database because the most important information for execution is stored as a linked list. It just seems hopelessly confusing and hard to interpret to save such information that way.

 

[suremarc]

A friend of mine used to work at NYSE. The lead software architect there (his name is Alexei Lebedev) created a system that generates what is essentially a zero-overhead in-memory database in C++. You define your data in the form of tabular relations, and it knows how to generate efficient code for algorithms and data structures from it. I have not used this system myself, but from what I have heard they use it extensively to encode highly cross-referenced data types as well as business entities (for example, other exchanges and their attributes, e.g. MIC’s), and case-driven testing.

An open-source implementation can be found at github.com/alexeilebedec/openacr. Even if you are not going to use it, I think the problem it attempts to solve is interesting and the author wrote a decent amount explaining his rationale. I recommend the read.

 

[Jarvis323]

Summary:: Most programming languages and environments store program source as text in source (and accompanying) files. But a program can be seen as a collection of relations between different objects, and thus these relations could also be stored in a data base. For example: every Variable has a relation to a Type, every function call implies a relation between caller and callee, etc. Has there ever been an attempt to store program source in the form of relations into a database?

Almost all (compiled or interpreted) programming langues store the program source in the form of a series of bytes (using an encoding like ASCII or UTF-8) into a text file, enforcing the grammer of the programming language using a parser (as part of the compilation process or interpretation of the source text).

But intrinsically a program can also be seen as a collection of relations between different objects, which can be stored as tuples in a relational database system.

For example:

• A program Variable needs to have a definition, which creates both a relation between the variable and a type, and also a location within another object (for example a function and/or module) in which the definition takes place.
• A function call creates a relation between the calling function or module and a called function, a relation between its return value and a type, and a relation between the function and the object or module in which it was defined.

Etc.

No programming language or environment (perhaps with the exception of a language/programming environment like smalltalk) however stores these relations as such, most languages store it as text in a source file.

The advantages of storing program source as relations in a database are multiple, like for instance:

• One can enforce strong typing rules for all objects.
• One can produce many usefull insights into the program source, like dependencies between objects (what function gets called by what function, what modules or objects use what variables, etc.).
• Compilation could be done piece wise (per object) as part of the editing proces (saving the object also pre-compiles it, and shows the errors encountered during compilation if any - source is stored anyway).
• Each variable or object name needs to be stored only once, and renaming objects would only take place at one point. Optionally, each programmer could name objects in their native language and script.
• The program itself can be stored in the form of an abstract syntax tree, simplyfying the process of creating object code and executable generation.
• Refactoring a program, or moving functions between modules, and retaining all dependency relations (like header files), could be done much simpler and less error prone then in a textual environment.
• Build/make information can be easily extracted from the relations already stored in the database, using the dependency relations already stored.
• Optionally also the versioning system itself could be implemented as part of the programming development system, also storing version information as relations into a database.
• For compatibility with the usual work environment and programming development tools, easy import and export functionality could be provided for such a programming development environment.

I guess it's probably not exactly what you're thinking about, but logic programming languages are based on sets of relations, with prolog being a good example.

the program logic is expressed in terms of relations, represented as facts and rules. A computation is initiated by running a query over these relations.

https://en.m.wikipedia.org/wiki/Prolog