Simulating an Experiment for a two state system

[bluechic92]

So I am so beyond stuck and my advisor told me that my progress is very slow. I made a post before about how I'm new to coding and everything, but now I kind of have to forget about the "learning" and make progress.

Does anyone have tips on understanding the Baum-Welch algorithm, forward-backward algorithm, and how to translate all of this into code? I found a really good lecture from Stanford's website (https://web.stanford.edu/~jurafsky/slp3/8.pdf ) and I understand the examples. I just have no clue how to translate this into code.

https://www.ph.tum.de/academics/org/labs/fopra/docs/userguide-82.en.pdf
"To identify conformational states of the sample we can plot the FRET efficiencies in a histogram. This approach is pictured in figure 6. Moreover, we can determine transition rates by measuring dwell times in the FRET traces. A dwell time is the period of time that a system remains in a given state. The distribution of dwell times is given by the rates of the underlying chemical reaction."

So I have written code that give me FRET traces with respect to time step. I was able to get a plot, and also two additional plot that gave me a histogram of the raw data and idealized data ( this plots only the frequency for the two states).

I have no idea how to write a code that measures how long something stays in one given state. I would then have to plot dwell time versus cumulative frequency and extract the transition rates ( which technically I know the correct value of).

In all honesty, my coding knowledge really is at beginner but the stuff required for this project seems advanced ( at least to me).

Thanks for any tips anyone!

 

[Nidum]

Pre written code modules are available for many standard mathematical and technical calculations .

Some of the common languages like C , Python and Fortran have extensive libraries of code modules .

See if you can find a module that does what you want and then try deconstructing it to learn how it works .

Google ' Code modules in (Python / C / whatever ) for scientific calculations ' first and then try a more specific search .

 

[Fooality]

Yeah, the secret of being an effective programmer is code as little as possible. The wiki page has a list of implementations in code, under section of that title
https://en.m.wikipedia.org/wiki/Baum–Welch_algorithm
Take one in a language you know, (or google around to find one) get it working, and then map the code to your math understanding, and figure out what you need to modify. This way you bypass all the time consuming coding mistakes.

 

[chiro]

Hey bluechic92.

You should probably look at something like R (an open source statistics platform), MATLAB, or find matrix and probability modeling libraries in some other language.

I would recommend R and look up the routines for Markov models.

The link to R is https://www.r-project.org/

In terms of code it will involve (most likely) the use of numerical techniques to solve implicit functions (including matrix based ones) or formulas to find some output as a function of finite specific instructions involving some arithmetic (including the use of matrix operations).

You have the conditions for the algorithm and the missing part is to build data structures, code in the operations and then write the decision statements (like if, for/while, do/loop and similar) that take the objects listed in the algorithm and be in the code itself.

You will have to code all functions (like a maximum function) separately or you can use something like R or MATLAB to make it easier.

 

[bluechic92]

Yeah, the secret of being an effective programmer is code as little as possible. The wiki page has a list of implementations in code, under section of that title
https://en.m.wikipedia.org/wiki/Baum–Welch_algorithm
Take one in a language you know, (or google around to find one) get it working, and then map the code to your math understanding, and figure out what you need to modify. This way you bypass all the time consuming coding mistakes.

I spent a lot of time on this wiki page! Thanks for the tip!

Pre written code modules are available for many standard mathematical and technical calculations .

Some of the common languages like C , Python and Fortran have extensive libraries of code modules .

See if you can find a module that does what you want and then try deconstructing it to learn how it works .

Google ' Code modules in (Python / C / whatever ) for scientific calculations ' first and then try a more specific search .

Sometimes I find reading other people's code to be extremely difficult, but I guess it's easier than trying to figure out how to write one. I think it's hard for me since I'm just starting out.

Hey bluechic92.

You should probably look at something like R (an open source statistics platform), MATLAB, or find matrix and probability modeling libraries in some other language.

I would recommend R and look up the routines for Markov models.

The link to R is https://www.r-project.org/

In terms of code it will involve (most likely) the use of numerical techniques to solve implicit functions (including matrix based ones) or formulas to find some output as a function of finite specific instructions involving some arithmetic (including the use of matrix operations).

You have the conditions for the algorithm and the missing part is to build data structures, code in the operations and then write the decision statements (like if, for/while, do/loop and similar) that take the objects listed in the algorithm and be in the code itself.

You will have to code all functions (like a maximum function) separately or you can use something like R or MATLAB to make it easier.

You are the second person to suggest R to me in the past couple of days! I think I will definitely have to check it out now. Thank you for the additional advice!

Also to anyone: Is this all supposed to be straight forward? I feel like this isn't that simple. My prof said this is straightforward...

 

[QuantumQuest]

As Nidum and Fooality point out, you need to use some prewritten code in whatever form this could be - modules, libraries of a programming language or ready-to-go code found on the net, and this is a necessity for a programmer to survive and not just a tip. It is more than certain that at the beginner level, code already written - particularly advanced, seems as magic, but it is not. It's a matter of a structured approach in solving problems and developing a mindset on how to translate this into code.

So, it is absolutely necessary, even if being a programmer is not your goal, to have a basic grasp of the absolute minimum of at least one programming language, in order to be in a position to write the "scaffolding" of a program, to use some modules or other prewritten code and last but not least, to be able to read and understand code written by others. There is no easy way or shortcut to this.

That said, you'll often find pieces of code, in which the author "uses his right hand to scratch his left ear" - as we say in my mother language, intentionally or not. And this also presupposes some decent understanding of coding in order to be resolved.

I think that sparing some time in a scheduled and systematic way to learn programming, will save you many hours of frustration and disappointment. After all, it's not useless in its own right. You'll definitely need it as a scientist and the pay-back will be huge.

 

[bluechic92]

As Nidum and Fooality point out, you need to use some prewritten code in whatever form this could be - modules, libraries of a programming language or ready-to-go code found on the net, and this is a necessity for a programmer to survive and not just a tip. It is more than certain that at the beginner level, code already written - particularly advanced, seems as magic, but it is not. It's a matter of a structured approach in solving problems and developing a mindset on how to translate this into code.

So, it is absolutely necessary, even if being a programmer is not your goal, to have a basic grasp of the absolute minimum of at least one programming language, in order to be in a position to write the "scaffolding" of a program, to use some modules or other prewritten code and last but not least, to be able to read and understand code written by others. There is no easy way or shortcut to this.

That said, you'll often find pieces of code, in which the author "uses his right hand to scratch his left ear" - as we say in my mother language, intentionally or not. And this also presupposes some decent understanding of coding in order to be resolved.

I think that sparing some time in a scheduled and systematic way to learn programming, will save you many hours of frustration and disappointment. After all, it's not useless in its own right. You'll definitely need it as a scientist and the pay-back will be huge.

I think this post gave you the wrong impression... I do care about learning how to program. At first I started learning C++ but then decided that Python might be an easier language to learn first. I consider myself a beginner and I am interested in developing a basic foundation. I am learning using a combination of books, Sololearn, Codeacademy, and getting help from experienced programmers.

I do not at all find reading other people's code to be that easy since I am still learning all the basics. I'm not frustrated that I can't figure this out, I am frustrated that I can't do it fast enough. My professor said I'm moving too slowly, even though I told him I'm just a beginner coder.

I have no interest in using short cuts to learn, but my advisor is not pleased that I don't have results faster. I wish I didn't need to sleep and eat so I can use 24 hours to learn+do my work. :/

 

[QuantumQuest]

I think this post gave you the wrong impression... I do care about learning how to program

That is good. I am software developer for about 15 years and my point was to encourage you to keep on learning programming.
Now, about programming languages I mainly use Java and C++, but Python is extremely good in many situations and in many ways. As I see, the combination of resources you use to learn coding is good enough but reading and analyzing code is a top priority in my opinion.

I do not at all find reading other people's code to be that easy since I am still learning all the basics.

I also pointed this out, giving you a warning about the prewritten code you find on the net.

My professor said I'm moving too slowly, even though I told him I'm just a beginner coder.

This is not something that can be done in an overnight fashion. It requires years to be done correctly.

 

[D H]

This is not something that can be done in an overnight fashion. It requires years to be done correctly.

A specific example: When one adds up the numbers, the Space Shuttle flight software was written at the excruciatingly slow rate of a line or two of code per week per person. (Requirements, test code, meetings galore, etc.: They don't. Only delivered code counts.)

We can do a bit better than this now by using modern programming techniques. (But not much better; one line of code per person per day remains the going rate for newly developed high quality, safety critical software.) We can do much, much better than this by reusing not writing software. Others have worked for decades to remove the kinks, and now much of this is open source. (Google "core flight software".)

The same applies to numerical programming. Roll your own and it will (a) take a long time, and (b) not work right. There is an immense amount of open source scientific software for C, C++, Fortran, Python, R, and Ruby (intentionally listed alphabetically). When one adds up the numbers, some of this scientific software was developed at the excruciatingly slow rate of a line or two of code per week per person. You get to use it for free. Well, not quite for free; their's always a hidden cost in reusing software developed elsewhere. But that cost is typically much lower than is the cost of rolling your own.

 

[chiro]

As for your question on straight-forwardness the best thing I can advise is to look up the routines concerning your modelling and see how much you need to merge them together to do what you need to do.

Simulating random variables is really easy in R (since it is built for statistics and probability modeling) and there are plenty of functions you can use to get a distribution for state changes (you just construct a new random variable that finds the distance between two similar events and this can be done in a simple for loop across the random vector).

If you are familiar with something like C/C++ it shouldn't take you that long if you know the distributional assumptions. Markov chains can be simulated as can any other random variable with all kinds of properties in R and you can do plots very easily and customize them very easily too.

 

[QuantumQuest]

A specific example: When one adds up the numbers, the Space Shuttle flight software was written at the excruciatingly slow rate of a line or two of code per week per person. (Requirements, test code, meetings galore, etc.: They don't. Only delivered code counts.)

We can do a bit better than this now by using modern programming techniques. (But not much better; one line of code per person per day remains the going rate for newly developed high quality, safety critical software.) We can do much, much better than this by reusing not writing software. Others have worked for decades to remove the kinks, and now much of this is open source. (Google "core flight software".)

The same applies to numerical programming. Roll your own and it will (a) take a long time, and (b) not work right. There is an immense amount of open source scientific software for C, C++, Fortran, Python, R, and Ruby (intentionally listed alphabetically). When one adds up the numbers, some of this scientific software was developed at the excruciatingly slow rate of a line or two of code per week per person. You get to use it for free. Well, not quite for free; their's always a hidden cost in reusing software developed elsewhere. But that cost is typically much lower than is the cost of rolling your own.

Agreed. All these are true and I made the same point about reusing software:

...you need to use some prewritten code in whatever form this could be - modules, libraries of a programming language or ready-to-go code found on the net, and this is a necessity for a programmer to survive and not just a tip.

.

That's for sure. We all do it, but the point is when and how. To be more specific, there has been grown a whole new "generation" of "programmers" that are nothing but copy-pasters. Now, while this is fine for the purposes of some homework - although I cannot overlook the morality issues of this, it is definitely not, for anyone working professionally in the code world. So, my point is essentially the development of the mindset of the good programmer, that is a time-consuming and tedious process. Many programmers learned their lesson from this, when the myriads of code lines they copy pasted, became their worst nightmare after a while, because simply they did not know exactly what they were doing.

So, all in all, there's no point to reuse code and pieces of software in general, without being first a decent programmer and I again stress out that I talk about professional coding and not some amateurish endeavors.

 

[D H]

To be more specific, there has been grown a whole new "generation" of "programmers" that are nothing but copy-pasters.

Agreed. We saw that with the initial rollout of healthcare.gov, which as far as I can tell was a badly done cut-and-paste job using mostly javascript and PHP.

 

[bluechic92]

I agree with all of you! I am interested in becoming better at coding and programming. I'm also interested in machine learning, but basics first.

I think this is just something I am slow at and need time to really work on. Unfortunately I don't have such a time because of my project. My advisor isn't very happy with me because of my slowness, and I really do care. I care about making progress, but I don't want to half ass it. I don't want to bull dang code by coping/pasting someone else's.

It's making feel demotivated and a bit dumb, honestly. I feel like he doesn't pay attention when I tell him that I am a beginner.

That's why I wish there was a magic way I could be faster. The advice I've been getting is to read/modify someone else's code. Which would be easier if I wasn't a beginner. It kind of seems simpler to learn how to write my own...

 

[Nidum]

It kind of seems simpler to learn how to write my own...

Let's have a go then .

What is your chosen programming language ?

Can you define the essential mathematical processes needed as a logical sequence of events ?

And can you make a flow chart ?

 

[bluechic92]

It kind of seems simpler to learn how to write my own...

Let's have a go then .

What is your chosen programming language ?

Can you define the essential mathematical processes needed as a logical sequence of events ?

And can you make a flow chart ?

Sorry I haven't responded!

I am learning Python in order to do this project.

Yes and yes to your next questions. Though translating into code is just something i am slow it. I am realizing that will need to make a set of transformation matrix and generate one as well. I am having an old friend help me!