=Fundamentals= Xreate is created to solve //the problem//(to allow easily and conveniently write an efficient and safe code) offering //the solution//(by combining largely opposite declarative and imperative approaches to cancel mutual limitations, using annotations to link them together). That's the unavoidable foundation, everything else is a subject of discussions. =Implemented, but not documented yet= ==1. Versions== Versions is a language construct to deal with mutable data. A short example: ``` name="Example 1" x{0} = {year = 1953, month = "March"}:: Date. x{1} = x{0} + {month = "February"} :: Date. //updates a single structure's field ``` `x[0]`, `x{1}` are different versions of the same variable. This is a simple trick with an idea that for all intents and purposes `x{0}`, `x{1}` behave like different variables but really aren't. All analyses treat them as different immutable variables and everything looks good. However, the compiler uses the same memory's address for them making this an update of a variable. It is a hint from a developer that the only one version of a variable should be available at any given time. The only analysis that knows the truth is Versions Analysis. Its task's to make sure that there is no expression that uses an outdated/unknown/unreachable variable's version. An another (counter)example: ``` COUNTEREXAMPLE, name="Example 2" x{0} = 8:: int. x{1} = x{0} + 10:: int. y = x{0} + x{1} :: int. ``` Versions Analysis builds a variables's liveliness graph to track situations when it's impossible to compute an expression due to the fact that it refers to an (possibly) outdated version or if it has lost a track of the current version due to branching, etc(Example 2). NOTE: Versions feature is in a very prototype state as of now and requires additional work. =Critical Path: Most important, but still not implemented aspects= The critical path includes pointers, memory management. mutable data(global variables), containers, strings, exceptions. They are expanded below. ==Pointers. Memory management== As of now, Xreate does not have pointers or overt memory management. Some directions: - Memory allocation. Automatically(with manual hints via annotations) tries to recognize and decide which variables where to allocate: -- Stack. -- Fixed size memory segment with possibly predefined addresses to allocate required memory really fast. -- Heap, for the cases where the reasoning can't offer anything better. - Garbage collection. Automatically(with manual hints via annotations) tries to recognize and decide which variables how to collect: -- Link life time of one variable `a` to another one `b`. Once `b` is collected, collect also `a`. -- Scope based lifetime. Note, that it's not necessary the scope where the variable in question is defined. It's easy to detect that scope/function is invoked in a loop, so choosing some other scope where it's explicitly allowed to take a moment to dispose of all the built up garbage would be better. -- References counting strategy, if no other strategy's applicable. ==Mutable data== Mutable data covered by Communication. Communication checks order in which mutable variables are read or written to detect value loss and other inconsistencies. Communication is still very briefly sketched prototype. In essence, pointers, variables' versions(described above) and Communication are crucial building blocks for the language. ==Containers and strings== Once pointers are done, it's possible to focus on Containers. Containers is an extension to automatically choose the most appropriate implementation out of a pool of existing ones, depending on the usage. The same goes for strings. For example, encountering foreign C function `strcmp(a, b)` the Containers reasoning should suggest C-type string(null terminated) as an implementation for `a, b` variables. On the other hand, if it encounters internal function `StrLen(a)` which requires(via an annotation) implementation to have already stored string size, it should infer something like a Pascal-type string(with its length stored in the first byte). Besides a usage patterns recognition, strings should have possibility to be configured by catching a developer's intents expressed via annotations, for example, `i_dont_want(cow)` (copy on write) or `i_dont_need(unicode)`. Note that annotations can express different modalities, a chosen implementation may(second annotation) or may not(first one) have undesired feature. As of now Containers reasoning handles which structures to make eager and which lazy ones. Basically this covers a whole range of different languages' constructs, techniques and approaches related to strings, laziness, generators, etc. ==Exceptions== Not implemented feature yet. More on this later. =Not implemented features yet. Do not belong to the Critical Path= More isolated, independent features of various importance that can be implemented(or not) in no particular order. ==Type system and type inference== Modern languages's type system is somewhat convoluted in order to take over some first/high order logic features. For example, phantom types are used to check static correctness of using validated/unvalidated data. In the case of Xreate this situation has to be rectified in order to draw a border between what is a responsibility of the hard-coded type system and what belongs to Transcend proper. For a start, Transcend in case of inability to prove correctness statically is able to assign a default strategy to check properties in question dynamically or make some adjustments to minimize a possible damage if a program is really incorrect. Second, Transcend able to work with preferences, choosing the most appropriate solution out of a number of possible candidates. Anyway, as of now basic [[ /d/syntax/#types | ADT type system ]] is implemented. ==Numbers and automatic conversion== The idea behind numerical types is to work on different levels of detailing: - `iN` e.g. `i32` denotes fully defined size. This automatically means the highest level of compiler warnings regarding conversions, overflows, etc. - `int` reserved as a placeholder for the compiler to come up with an appropriate integral type staying reasonably efficient. As of now hard aliased to `i32`. - `num` reserved as a placeholder for the compiler to come up with an appropriate integral or float type if a developer can't assign anything better. As of now hard aliased to `i32`. ==Contracts and Diagnostics== Transcend is really easy can be used to statically check software contracts and signal warnings straight to the point. An example: ``` //a harmful function that has multiple bugs getNumber = function:: int; bug(4379); bug(12990) { 0 } client = function:: int; contract(call_only_good_functions) { getNumber() } ``` Transcend rules to support contract `call_only_good_functions` presented below: ``` bind_func(X, bad_function):- bind_func(X, bug(_)). //marks bugged function as bad_function warning("Contract violation: call_only_good_functions") :- //looks for the contract violations cfa_call(Scope, BadFn); fn_scope(GoodFn, Scope); bind_func(BadFn, bad_function); bind_func(GoodFn, contract(call_only_good_functions)). ``` This excerpt prohibits functions annotated `contract(call_only_good_functions)` to call functions with identified bugs. ==Supervision== Similar to Contract and Diagnostics(described above) but on a level of an environment where a code is compiled or executed. Defines a contract between an environment(an OS, a machine, a local network) and a compiled or executed software, e.g. ensures that the software can't use a web camera or open local files. In case of an already compiled software it has to have a trusted compiler issued certificate to confirm that it's indeed, let's say, does not open any local files and the certificate is examined to check compliance with the environment's rules instead of the actual source code. This does not require any special support from an underlying OS and does not require sandboxing which is always imply a certain performance degradation. Also Transformer technique(described below) can be used to remove inappropriate for this environment parts of a code. =="Why" subsytem== Due to the fact that everything is reordered, transformed, etc, programming in Xreate can be perceived as complicated. One way to alleviate this is to reserve a functionality to answer queries about the reasoner's work, such as decisions, solutions. In other words, to help a developer find out several information pieces at any time: - All annotations attached(manually/automatically) to a variable. - All annotations attached(manually/automatically) to a code scope/function/module. - What is a chosen implementation for this or that variable. - What is a decision regarding this or that aspect(virtualization, etc). - Why a decision is this and not that. ==Code Transformer(self modifying code at a runtime) == The compiler supplies data for a Transcend script [[ http://xreate.org/diffusion/XR/browse/master/scripts/cfa/payload.lp | payload.lp ]] to build a Dominator Tree that represents an order of execution of different parts of a program. An example: ``` partFirst = function:: int {0} //the first part of a program partSecond = function:: int {0} //the second part of a program main = function:: int; entry { seq { partFirst() } { partSecond() } } ``` In this example `partSecond` is executed strictly after `partFirst` and the control flow never visits `partFirst` again. By looking at a dominator tree it's possible to recognize situations when some code after execution is not used any more. This gives a possibility(at an appropriate time) to remove a useless code that already fulfills its purposes. Also this can be used by Supervision(described above) to remove inappropriate for a given environment code parts without recompilation. ==Reactivity and Tracking Control== More on this later ==Time machine== A source code can be reordered in two ways: either executed some time in the future(sending the code to the future) or some time in the past(sending the code to the past). Let's consider an example: ``` flagDoSaveFile = if(fileExists("/some/file") :: bool { answer = ask("A file already exists Do you really want to overwrite") :: bool. answer } else { true } if(flagDoSaveFile):: Result { saveFile("/some/file") } ``` This code tries to save a file and asks a user to confirm if the file already exists. Suppose we let software to work overnight and now want all the file operations to be silent. In order to do that, we need to adapt the code to ask everything at the beginning. The result can look like this: ``` { context:: mode(interactive). //this is the time to ask everything while we can askAllQuestion(). } ... { context:: mode(silent). //Alas. it's a silent mode, we can ask questions no more! copyFiles(). } ``` By using Time machine is possible to: - Recognize that function `ask` requires interactive mode by adding annotation like this to the function header: `allowed_time(mode(interactive))`. - Recognize when a function is invoked at the wrong time frame, based on annotations. - Reorder a code in such a way so that functions are executed at appropriate moments of a time. At the time of actual invocation it just returns already computed result. In other words, it's a way to support Incremental Programming - technique to adapt a code by adding new functionality to the end(or to the beginning) without modifying existing code, thus simplifying versioning, testing, catching regressions, etc. =Language's drawbacks and disadvantages= Peculiarities that need special care to conceal and mitigate: - Indefinite compilation time. The compiler clearly trades off execution efficiency against compilation time. Compilation time is cheaper in a sense that software is compiled once on a developer's machine but run multiple times on multiple users' machines. However, Transcend being just designation of a SAT logic solver backend has the same inefficiency as always. Thus usual AI measures apply. Such as limited depth of a searched space, etc. - Everything is rewritten, reordered, transformed, adapted and silently decided. A whole lot of the different things done under the hood. Great amount of added complexity. This can be targeted by "Why" subsystem to allow a developer to know what's going on at any given moment. - Unexpected decision changes. =Language's distinctive possible applications and selling points= - Convergent and spread out software. Software with the same/similar functionality can be packed in different forms: mobile application, desktop, self-hosted locally, cloud based, single/multi user interface, etc. Xreate can be useful to handle this on 2 levels: -- GUI. A Full fledged logic solver is particularly suited to process declarative data such as GUI. By writing GUI as Transcend data it's possible to transform it into several forms for a desktop, mobile, etc interface. Also a static analysis can be run against GUI to check an interaction sequence to ensure, for example, that specific GUI states(such as the main screen) are always reachable no matter what user clicks. FOL Logic solver can possibly substitute whole range of endlessly multiplying technologies such as HTML/CSS/JS/REACT, XML/XSLT/XPATH/XQUERY, whatever. -- Virtualization and Supervision. Virtualization reasoning is able to apply different virtualization strategies for different code parts. Those strategies are as lightweight as possible just enough to ensure required safety level. On the other hand, Supervision can optimize and provide more safety of software for a given platform without recompilation. - Combinatorial libraries. Software factories. In essence Transcend picks up appropriate specializations from a range of available ones while building client software. This can be conceptualized as a software factory that builds client software from isolated blocks that satisfy the given requirements. Containers reasoning tries to decide which container's implementation to choose based on how the container is used in the code. The can be a foundation for a full fledged database that can be compiled just for a single client or a set of clients in a particular form such as relational DB, document oriented, graph oriented, embedded or client-server based, with or without transactions based solely on client's needs, queries and processed data. - Low-end hardware. IoT. Low-end hardware has hard computational power limit due to electricity consumption and battery life. Most developers would be glad to code using convenient languages like Python but performance issues are not well suited for low end hardware. Xreate can be possible a solution blending features of different languages. - Software that can not be written by professional developers. Scientific software and massive computations. Scientific software is written by scientists that have no time(and desire) to learn how to code effectively. Thus they use very simple script languages to perform AI related and other massive computations that suffers from severe performance issues. That can be also an stimulus to develop Xreate. =License= The compiler is under Mozilla's MPL 2.0. This means that the code can be combined with a GPL'ed code as well as with a proprietary code(under condition that MPL'ed and proprietary parts reside in different files). The intention is to make the compiler clearly FOSS project with an ability to use it with proprietary plugins, extensions, IDE, etc as a very remote but possible option. To make this possible, the contributors are free to assign their own licenses for their respective parts as long as license in question is at least as permissive as MPL(which is expected) or MIT or something of this sort. Anyway, not a single stroke will be used without a fully informed contributor's consent should any controversies arise. GPL license: due to the reasons above, a source code of the core components may not be GPL'ed. However there are still legitimate uses for GPL(or other restrictive licenses). The compiler has two distinctive backends(or rather three): - Brute byte code generation backend. Class `LLVMLayer` (and all the files in `compilation` folder) - are responsible for producing a byte code (currently LLVM). If someone adds an interface to a different backend , notably GCC, it surely may have a different, respective backend's compatible license. - A logic solver backend. Class `TranscendLayer`(and files in `analysis` folder) - compile logic programs for an external solver(currently Clasp) to process and send a solution back. There are many different solvers out there, so the same considerations apply. - A foreign interface backend - an interface to different languages. Currently it's `clang` to use external C functions in Xreate. Different languages' bridges may require different licenses.

=Fundamentals= Xreate is created to solve //the problem//(to allow easily and conveniently write an efficient and safe code) offering //the solution//(by combining largely opposite declarative and imperative approaches to cancel mutual limitations, using annotations to link them together). That's the unavoidable foundation, everything else is a subject of discussions. =Entry level participation= - Documentation: proofread, grammar and spelling, broken links, checking installation instructions, examples, etc. - Installation: as of now OpenSuse Linux is supported. Installation instructions for other distributions. - Language: review and streamline the language's syntax, immediate show-stoppers, suggestions, etc. Proposals of syntax sugar to simplify frequently used the language's constructs. - Compiler: code cleanup, memory leaks, porting to other platforms. =Critical Path: Most important, but still not implemented aspects= The critical path includes pointers, memory management. mutable data(global variables), containers, strings, exceptions. They are expanded below. ==Pointers. Memory management== As of now, Xreate does not have pointers or overt memory management. Some directions: - Memory allocation. Automatically(with manual hints via annotations) tries to recognize and decide which variables where to allocate: -- Stack. -- Fixed size memory segment with possibly predefined addresses to allocate required memory really fast. -- Heap, for the cases where the reasoning can't offer anything better. - Garbage collection. Automatically(with manual hints via annotations) tries to recognize and decide which variables how to collect: -- Link life time of one variable `a` to another one `b`. Once `b` is collected, collect also `a`. -- Scope based lifetime. Note, that it's not necessary the scope where the variable in question is defined. It's easy to detect that scope/function is invoked in a loop, so choosing some other scope where it's explicitly allowed to take a moment to dispose of all the built up garbage would be better. -- References counting strategy, if no other strategy's applicable. ==Mutable data== Mutable data covered by Communication. Communication checks order in which mutable variables are read or written to detect value loss and other inconsistencies. Communication is still very briefly sketched prototype. In essence, pointers, variables' versions(described above) and Communication are crucial building blocks for the language. ==Containers, strings, files== Once pointers are done, it's possible to focus on Containers. Containers is an extension to automatically choose the most appropriate implementation out of a pool of existing ones, depending on the usage. The same goes for strings. For example, encountering foreign C function `strcmp(a, b)` the Containers reasoning should suggest C-type string(null terminated) as an implementation for `a, b` variables. On the other hand, if it encounters internal function `StrLen(a)` which requires(via an annotation) implementation to have already stored string size, it should infer something like a Pascal-type string(with its length stored in the first byte). Besides a usage patterns recognition, strings should have possibility to be configured by catching a developer's intents expressed via annotations, for example, `i_dont_want(cow)` (copy on write) or `i_dont_need(unicode)`. Note that annotations can express different modalities, a chosen implementation may(second annotation) or may not(first one) have undesired feature. As of now Containers reasoning handles which structures to make eager and which lazy ones. Basically this covers a whole range of different languages' constructs, techniques and approaches related to strings, laziness, generators, etc. Files as yet another aspect that can be present as containers. For example, sorting content of a file-container should automatically infer appropriate, namely external sorting algorithm's implementation rather then default in memory sorting. ==Exceptions== Not implemented feature yet. More on this later. =Not implemented features yet. Do not belong to the Critical Path= More isolated, independent features of various importance that can be implemented(or not) in no particular order. ==Type system and type inference== Modern languages's type system is somewhat convoluted in order to take over some first/high order logic features. For example, phantom types are used to check static correctness of using validated/unvalidated data. In the case of Xreate this situation has to be rectified in order to draw a border between what is a responsibility of the hard-coded type system and what belongs to Transcend proper. For a start, Transcend in case of inability to prove correctness statically is able to assign a default strategy to check properties in question dynamically or make some adjustments to minimize a possible damage if a program is really incorrect. Second, Transcend able to work with preferences, choosing the most appropriate solution out of a number of possible candidates. Anyway, as of now basic [[ /d/syntax/#types | ADT type system ]] is implemented. ==Numbers and automatic conversion== The idea behind numerical types is to work on different levels of detailing: - `iN` e.g. `i32` denotes fully defined size. This automatically means the highest level of compiler warnings regarding conversions, overflows, etc. - `int` reserved as a placeholder for the compiler to come up with an appropriate integral type staying reasonably efficient. As of now hard aliased to `i32`. - `num` reserved as a placeholder for the compiler to come up with an appropriate integral or float type if a developer can't assign anything better. As of now hard aliased to `i32`. ==Contracts and Diagnostics== Transcend is really easy can be used to statically check software contracts and signal warnings straight to the point. An example: ``` //a harmful function that has multiple bugs getNumber = function:: int; bug(4379); bug(12990) { 0 } client = function:: int; contract(call_only_good_functions) { getNumber() } ``` Transcend rules to support contract `call_only_good_functions` presented below: ``` bind_func(X, bad_function):- bind_func(X, bug(_)). //marks bugged function as bad_function warning("Contract violation: call_only_good_functions") :- //looks for the contract violations cfa_call(Scope, BadFn); fn_scope(GoodFn, Scope); bind_func(BadFn, bad_function); bind_func(GoodFn, contract(call_only_good_functions)). ``` This excerpt prohibits functions annotated `contract(call_only_good_functions)` to call functions with identified bugs. ==Supervision== Similar to Contract and Diagnostics(described above) but on a level of an environment where code is compiled or executed. Defines a contract between an environment(an OS, a machine or a local network) and compiled or executed software, e.g. it ensures that the software can't use a web camera or open local files. In case of already compiled software it has to have a trusted compiler issued certificate to confirm that it's indeed, let's say, does not try to open any local files and the certificate is examined to check compliance with the environment's rules instead of the actual source code. This does not require any special support from an underlying OS and does not require sandboxing which is always imply a certain performance degradation. Also Transformer technique(described below) can be used to remove inappropriate for this environment parts of code. =="Why" subsytem== Due to the fact that everything is reordered, transformed, etc, programming in Xreate can be perceived as complicated. One way to alleviate this is to reserve a functionality to answer queries about the reasoner's work, such as decisions, solutions. In other words, to help a developer find out several information pieces at any time: - All annotations attached(manually/automatically) to a variable. - All annotations attached(manually/automatically) to a code scope/function/module. - What is a chosen implementation for this or that variable. - What is a decision regarding this or that aspect(virtualization, etc). - Why a decision is this and not that. ==Code Transformer(self modifying code at runtime) == The compiler supplies data for a Transcend script [[ http://xreate.org/diffusion/XR/browse/master/scripts/cfa/payload.lp | payload.lp ]] to build a Dominator Tree that represents an order of execution of different parts of a program. An example: ``` partFirst = function:: int {0} //the first part of a program partSecond = function:: int {0} //the second part of a program main = function:: int; entry { seq { partFirst() } { partSecond() } } ``` In this example `partSecond` is executed strictly after `partFirst` and the control flow never visits `partFirst` again. By looking at a dominator tree it's possible to recognize situations when some code after execution is not used any more. This gives a possibility(at an appropriate time) to remove a useless code that already fulfills its purposes. Also this can be used by Supervision(described above) to remove inappropriate for a given environment code parts without recompilation. ==Reactivity and Tracking Control== More on this later ==Time machine== A source code can be reordered in two ways: either executed some time in the future(sending the code to the future) or some time in the past(sending the code to the past). Let's consider an example: ``` flagDoSaveFile = if(fileExists("/some/file") :: bool { answer = ask("A file already exists Do you really want to overwrite") :: bool. answer } else { true } if(flagDoSaveFile):: Result { saveFile("/some/file") } ``` This code tries to save a file and asks a user to confirm if the file already exists. Suppose we let software to work overnight and now want all the file operations to be silent. In order to do that, we need to adapt the code to ask everything at the beginning. The result can look like this: ``` { context:: mode(interactive). //this is the time to ask everything while we can askAllQuestion(). } ... { context:: mode(silent). //Alas. it's a silent mode, we can ask questions no more! copyFiles(). } ``` By using Time machine is possible to: - Recognize that function `ask` requires interactive mode by adding annotation like this to the function header: `allowed_time(mode(interactive))`. - Recognize when a function is invoked at the wrong time frame, based on annotations. - Reorder a code in such a way so that functions are executed at appropriate moments of a time. At the time of actual invocation it just returns already computed result. In other words, it's a way to support Incremental Programming - technique to adapt a code by adding new functionality to the end(or to the beginning) without modifying existing code, thus simplifying versioning, testing, catching regressions, etc. =Language's drawbacks and disadvantages= Peculiarities that need special care to conceal and mitigate: - Compilation time. The compiler clearly trades off execution efficiency against compilation time. Compilation time is cheaper in a sense that software is compiled once on a developer's machine but run multiple times on multiple users' machines. However, Transcend being just designation of a SAT logic solver backend has the same inefficiency as always. Thus usual AI measures apply. Such as limited depth of a searched space, etc. - Everything is rewritten, reordered, transformed, adapted and silently decided. A whole lot of the different things done under the hood. Great amount of added complexity. This can be targeted by "Why" subsystem to allow a developer to know what's going on at any given moment. - Unexpected decision changes. =Language's distinctive possible applications and selling points= - Convergent and spread out software. Software with the same/similar functionality can be packed in different forms: mobile application, desktop, self-hosted locally, cloud based, single/multi user interface, etc. Xreate can be useful to handle this on 2 levels: -- GUI. A Full fledged logic solver is particularly suited to process declarative data such as GUI. By writing GUI as Transcend data it's possible to transform it into several forms for a desktop, mobile, etc interface. Also a static analysis can be run against GUI to check an interaction sequence to ensure, for example, that specific GUI states(such as the main screen) are always reachable no matter what user clicks. FOL Logic solver can possibly substitute whole range of endlessly multiplying technologies such as HTML/CSS/JS/REACT, XML/XSLT/XPATH/XQUERY, whatever. -- Virtualization and Supervision. Virtualization reasoning is able to apply different virtualization strategies for different code parts. Those strategies are as lightweight as possible just enough to ensure required safety level. On the other hand, Supervision can optimize and provide more safety of software for a given platform without recompilation. - Combinatorial libraries. Software factories. In essence Transcend picks up appropriate specializations from a range of available ones while building client software. This can be conceptualized as a software factory that builds client software from isolated blocks that satisfy the given requirements. Containers reasoning tries to decide which container's implementation to choose based on how the container is used in the code. The can be a foundation for a full fledged database that can be compiled just for a single client or a set of clients in a particular form such as relational DB, document oriented, graph oriented, embedded or client-server based, with or without transactions based solely on client's needs, queries and processed data. - Low-end hardware. IoT. Low-end hardware has hard computational power limit due to electricity consumption and battery life. Most developers would be glad to code using convenient languages like Python but performance issues are not well suited for low end hardware. Xreate can be possible a solution blending features of different languages. - Software that can not be written by professional developers. Scientific software and massive computations. Scientific software is written by scientists that have no time(and desire) to learn how to code effectively. Thus they use very simple script languages to perform AI related and other massive computations that suffer from severe performance issues. That can be also a stimulus to develop language like Xreate targeting this niche. =License= The compiler is under Mozilla's MPL 2.0. This means that the code can be combined with a GPL'ed code as well as with a proprietary code(under condition that MPL'ed and proprietary parts reside in different files). The intention is to make the compiler clearly FOSS project with an ability to use it with proprietary plugins, extensions, IDE, etc as a very remote but possible option. To make this possible, the contributors are free to assign their own licenses for their respective parts as long as license in question is at least as permissive as MPL(which is expected) or MIT or something of this sort. Anyway, not a single stroke will be used without a fully informed contributor's consent should any controversies arise. GPL license: due to the reasons above, a source code of the core components may not be GPL'ed. However there are still legitimate uses for GPL(or other restrictive licenses). The compiler has two distinctive backends(or rather three): - Brute byte code generation backend. Class `LLVMLayer` (and all the files in `compilation` folder) - are responsible for producing a byte code (currently LLVM). If someone adds an interface to a different backend , notably GCC, it surely may have a different, respective backend's compatible license. - A logic solver backend. Class `TranscendLayer`(and files in `analysis` folder) - compile logic programs for an external solver(currently Clasp) to process and send a solution back. There are many different solvers out there, so the same considerations apply. - A foreign interface backend - an interface to different languages. Currently it's `clang` to use external C functions in Xreate. Different languages' bridges may require different licenses.

=Fundamentals= Xreate is created to solve //the problem//(to allow easily and conveniently write an efficient and safe code) offering //the solution//(by combining largely opposite declarative and imperative approaches to cancel mutual limitations, using annotations to link them together). That's the unavoidable foundation, everything else is a subject of discussions. =Implemented, but not documented yet==Entry level participation= ==1. Versions== Versions is a language construct to deal with mutable data. A short example: ``` name="Example 1" x{0} = {year = 1953, month = "March"}:: Date. x{1} = x{0} + {month = "February"} :: Date. //updates a single structure's field ``` `x[0]`, `x{1}` are different versions of the same variable. This is a simple trick with an idea that for all intents and purposes `x{0}`, `x{1}` behave like different variables but really aren't. All analyses treat them as different immutable variables and everything looks good. However, the compiler uses the same memory's address for them making this an update of a variable.- Documentation: proofread, It is a hint from a developer that the only one version of a variable should be available at any given time.grammar and spelling, The only analysis that knows the truth is Versions Analysis.broken links, Its task's to make sure that there is no expression that uses an outdated/unknown/unreachable variable's version.checking installation instructions, An another (counter)example: ``` COUNTEREXAMPLEs, name="Example 2"etc. x{0} = 8:: int. x{1} = x{0} + 10:: int. y = x{0} + x{1} :: int - Installation: as of now OpenSuse Linux is supported. Installation instructions for other distributions. ``` - Language: review and streamline the language's syntax, immediate show-stoppers, suggestions, etc. Proposals of syntax sugar to simplify frequently used the language's constructs. Versions Analysis builds a variables's liveliness graph to track situations when it's impossible to compute an expression due to the fact that it refers to an (possibly) outdated version or if it has lost a track of the current version due to branching - Compiler: code cleanup, etc(Example 2). NOTE: Versions feature is in a very prototype state as of now and requires additional workmemory leaks, porting to other platforms. =Critical Path: Most important, but still not implemented aspects= The critical path includes pointers, memory management. mutable data(global variables), containers, strings, exceptions. They are expanded below. ==Pointers. Memory management== As of now, Xreate does not have pointers or overt memory management. Some directions: - Memory allocation. Automatically(with manual hints via annotations) tries to recognize and decide which variables where to allocate: -- Stack. -- Fixed size memory segment with possibly predefined addresses to allocate required memory really fast. -- Heap, for the cases where the reasoning can't offer anything better. - Garbage collection. Automatically(with manual hints via annotations) tries to recognize and decide which variables how to collect: -- Link life time of one variable `a` to another one `b`. Once `b` is collected, collect also `a`. -- Scope based lifetime. Note, that it's not necessary the scope where the variable in question is defined. It's easy to detect that scope/function is invoked in a loop, so choosing some other scope where it's explicitly allowed to take a moment to dispose of all the built up garbage would be better. -- References counting strategy, if no other strategy's applicable. ==Mutable data== Mutable data covered by Communication. Communication checks order in which mutable variables are read or written to detect value loss and other inconsistencies. Communication is still very briefly sketched prototype. In essence, pointers, variables' versions(described above) and Communication are crucial building blocks for the language. ==Containers and, strings==s, files== Once pointers are done, it's possible to focus on Containers. Containers is an extension to automatically choose the most appropriate implementation out of a pool of existing ones, depending on the usage. The same goes for strings. For example, encountering foreign C function `strcmp(a, b)` the Containers reasoning should suggest C-type string(null terminated) as an implementation for `a, b` variables. On the other hand, if it encounters internal function `StrLen(a)` which requires(via an annotation) implementation to have already stored string size, it should infer something like a Pascal-type string(with its length stored in the first byte). Besides a usage patterns recognition, strings should have possibility to be configured by catching a developer's intents expressed via annotations, for example, `i_dont_want(cow)` (copy on write) or `i_dont_need(unicode)`. Note that annotations can express different modalities, a chosen implementation may(second annotation) or may not(first one) have undesired feature. As of now Containers reasoning handles which structures to make eager and which lazy ones. Basically this covers a whole range of different languages' constructs, techniques and approaches related to strings, laziness, generators, etc. Files as yet another aspect that can be present as containers. For example, sorting content of a file-container should automatically infer appropriate, namely external sorting algorithm's implementation rather then default in memory sorting. ==Exceptions== Not implemented feature yet. More on this later. =Not implemented features yet. Do not belong to the Critical Path= More isolated, independent features of various importance that can be implemented(or not) in no particular order. ==Type system and type inference== Modern languages's type system is somewhat convoluted in order to take over some first/high order logic features. For example, phantom types are used to check static correctness of using validated/unvalidated data. In the case of Xreate this situation has to be rectified in order to draw a border between what is a responsibility of the hard-coded type system and what belongs to Transcend proper. For a start, Transcend in case of inability to prove correctness statically is able to assign a default strategy to check properties in question dynamically or make some adjustments to minimize a possible damage if a program is really incorrect. Second, Transcend able to work with preferences, choosing the most appropriate solution out of a number of possible candidates. Anyway, as of now basic [[ /d/syntax/#types | ADT type system ]] is implemented. ==Numbers and automatic conversion== The idea behind numerical types is to work on different levels of detailing: - `iN` e.g. `i32` denotes fully defined size. This automatically means the highest level of compiler warnings regarding conversions, overflows, etc. - `int` reserved as a placeholder for the compiler to come up with an appropriate integral type staying reasonably efficient. As of now hard aliased to `i32`. - `num` reserved as a placeholder for the compiler to come up with an appropriate integral or float type if a developer can't assign anything better. As of now hard aliased to `i32`. ==Contracts and Diagnostics== Transcend is really easy can be used to statically check software contracts and signal warnings straight to the point. An example: ``` //a harmful function that has multiple bugs getNumber = function:: int; bug(4379); bug(12990) { 0 } client = function:: int; contract(call_only_good_functions) { getNumber() } ``` Transcend rules to support contract `call_only_good_functions` presented below: ``` bind_func(X, bad_function):- bind_func(X, bug(_)). //marks bugged function as bad_function warning("Contract violation: call_only_good_functions") :- //looks for the contract violations cfa_call(Scope, BadFn); fn_scope(GoodFn, Scope); bind_func(BadFn, bad_function); bind_func(GoodFn, contract(call_only_good_functions)). ``` This excerpt prohibits functions annotated `contract(call_only_good_functions)` to call functions with identified bugs. ==Supervision== Similar to Contract and Diagnostics(described above) but on a level of an environment where a code is compiled or executed. Defines a contract between an environment(an OS, a machine, aa machine or a local network) and a compiled or executed software, e.g. it ensures that the software can't use a web camera or open local files. In case of an already compiled software it has to have a trusted compiler issued certificate to confirm that it's indeed, let's say, does not try to open any local files and the certificate is examined to check compliance with the environment's rules instead of the actual source code. This does not require any special support from an underlying OS and does not require sandboxing which is always imply a certain performance degradation. Also Transformer technique(described below) can be used to remove inappropriate for this environment parts of a code. =="Why" subsytem== Due to the fact that everything is reordered, transformed, etc, programming in Xreate can be perceived as complicated. One way to alleviate this is to reserve a functionality to answer queries about the reasoner's work, such as decisions, solutions. In other words, to help a developer find out several information pieces at any time: - All annotations attached(manually/automatically) to a variable. - All annotations attached(manually/automatically) to a code scope/function/module. - What is a chosen implementation for this or that variable. - What is a decision regarding this or that aspect(virtualization, etc). - Why a decision is this and not that. ==Code Transformer(self modifying code at a runtime) == The compiler supplies data for a Transcend script [[ http://xreate.org/diffusion/XR/browse/master/scripts/cfa/payload.lp | payload.lp ]] to build a Dominator Tree that represents an order of execution of different parts of a program. An example: ``` partFirst = function:: int {0} //the first part of a program partSecond = function:: int {0} //the second part of a program main = function:: int; entry { seq { partFirst() } { partSecond() } } ``` In this example `partSecond` is executed strictly after `partFirst` and the control flow never visits `partFirst` again. By looking at a dominator tree it's possible to recognize situations when some code after execution is not used any more. This gives a possibility(at an appropriate time) to remove a useless code that already fulfills its purposes. Also this can be used by Supervision(described above) to remove inappropriate for a given environment code parts without recompilation. ==Reactivity and Tracking Control== More on this later ==Time machine== A source code can be reordered in two ways: either executed some time in the future(sending the code to the future) or some time in the past(sending the code to the past). Let's consider an example: ``` flagDoSaveFile = if(fileExists("/some/file") :: bool { answer = ask("A file already exists Do you really want to overwrite") :: bool. answer } else { true } if(flagDoSaveFile):: Result { saveFile("/some/file") } ``` This code tries to save a file and asks a user to confirm if the file already exists. Suppose we let software to work overnight and now want all the file operations to be silent. In order to do that, we need to adapt the code to ask everything at the beginning. The result can look like this: ``` { context:: mode(interactive). //this is the time to ask everything while we can askAllQuestion(). } ... { context:: mode(silent). //Alas. it's a silent mode, we can ask questions no more! copyFiles(). } ``` By using Time machine is possible to: - Recognize that function `ask` requires interactive mode by adding annotation like this to the function header: `allowed_time(mode(interactive))`. - Recognize when a function is invoked at the wrong time frame, based on annotations. - Reorder a code in such a way so that functions are executed at appropriate moments of a time. At the time of actual invocation it just returns already computed result. In other words, it's a way to support Incremental Programming - technique to adapt a code by adding new functionality to the end(or to the beginning) without modifying existing code, thus simplifying versioning, testing, catching regressions, etc. =Language's drawbacks and disadvantages= Peculiarities that need special care to conceal and mitigate: - Indefinite c- Compilation time. The compiler clearly trades off execution efficiency against compilation time. Compilation time is cheaper in a sense that software is compiled once on a developer's machine but run multiple times on multiple users' machines. However, Transcend being just designation of a SAT logic solver backend has the same inefficiency as always. Thus usual AI measures apply. Such as limited depth of a searched space, etc. - Everything is rewritten, reordered, transformed, adapted and silently decided. A whole lot of the different things done under the hood. Great amount of added complexity. This can be targeted by "Why" subsystem to allow a developer to know what's going on at any given moment. - Unexpected decision changes. =Language's distinctive possible applications and selling points= - Convergent and spread out software. Software with the same/similar functionality can be packed in different forms: mobile application, desktop, self-hosted locally, cloud based, single/multi user interface, etc. Xreate can be useful to handle this on 2 levels: -- GUI. A Full fledged logic solver is particularly suited to process declarative data such as GUI. By writing GUI as Transcend data it's possible to transform it into several forms for a desktop, mobile, etc interface. Also a static analysis can be run against GUI to check an interaction sequence to ensure, for example, that specific GUI states(such as the main screen) are always reachable no matter what user clicks. FOL Logic solver can possibly substitute whole range of endlessly multiplying technologies such as HTML/CSS/JS/REACT, XML/XSLT/XPATH/XQUERY, whatever. -- Virtualization and Supervision. Virtualization reasoning is able to apply different virtualization strategies for different code parts. Those strategies are as lightweight as possible just enough to ensure required safety level. On the other hand, Supervision can optimize and provide more safety of software for a given platform without recompilation. - Combinatorial libraries. Software factories. In essence Transcend picks up appropriate specializations from a range of available ones while building client software. This can be conceptualized as a software factory that builds client software from isolated blocks that satisfy the given requirements. Containers reasoning tries to decide which container's implementation to choose based on how the container is used in the code. The can be a foundation for a full fledged database that can be compiled just for a single client or a set of clients in a particular form such as relational DB, document oriented, graph oriented, embedded or client-server based, with or without transactions based solely on client's needs, queries and processed data. - Low-end hardware. IoT. Low-end hardware has hard computational power limit due to electricity consumption and battery life. Most developers would be glad to code using convenient languages like Python but performance issues are not well suited for low end hardware. Xreate can be possible a solution blending features of different languages. - Software that can not be written by professional developers. Scientific software and massive computations. Scientific software is written by scientists that have no time(and desire) to learn how to code effectively. Thus they use very simple script languages to perform AI related and other massive computations that suffers from severe performance issues. That can be also ana stimulus to develop Xreatelanguage like Xreate targeting this niche. =License= The compiler is under Mozilla's MPL 2.0. This means that the code can be combined with a GPL'ed code as well as with a proprietary code(under condition that MPL'ed and proprietary parts reside in different files). The intention is to make the compiler clearly FOSS project with an ability to use it with proprietary plugins, extensions, IDE, etc as a very remote but possible option. To make this possible, the contributors are free to assign their own licenses for their respective parts as long as license in question is at least as permissive as MPL(which is expected) or MIT or something of this sort. Anyway, not a single stroke will be used without a fully informed contributor's consent should any controversies arise. GPL license: due to the reasons above, a source code of the core components may not be GPL'ed. However there are still legitimate uses for GPL(or other restrictive licenses). The compiler has two distinctive backends(or rather three): - Brute byte code generation backend. Class `LLVMLayer` (and all the files in `compilation` folder) - are responsible for producing a byte code (currently LLVM). If someone adds an interface to a different backend , notably GCC, it surely may have a different, respective backend's compatible license. - A logic solver backend. Class `TranscendLayer`(and files in `analysis` folder) - compile logic programs for an external solver(currently Clasp) to process and send a solution back. There are many different solvers out there, so the same considerations apply. - A foreign interface backend - an interface to different languages. Currently it's `clang` to use external C functions in Xreate. Different languages' bridges may require different licenses.