Distributed systems

Fundamentals: goals, communication, coordination, programming errors
Parallel decomposition: data and functional paralelism, pipeline, scalability
Communication and coordination: shared-memory systems (shared memory, cache coherence, mutual exclusion, barriers, conditional variables, semaphores), distributed-memory systems (message passing, point-to-point versus multicast, latency and throughput)
Parallel algorithms, analysis and programming: concepts and examples (embarisingly parallel algorithms, divide and conquer, map reduce, master slave), analyis (speedup and Amdahl's law, scalability)
Parallel architectures: Flynn's taxonomy, shared-memory systems (SIMD, UMA, NUMA), distributed-memory systems (cluster, grid, cloud), graphic processing units, modern coprocessors and heterogeneous systems, topologies (buses and interconnects)
Parallel performance: load balancing, scheduling and contention, communication overhead, cache effects, spatial and temporal datal locality, energy efficiency
Theoretical models of parallel computing: formal models of parallel computation (PRAM, BSP), formal models of message passing (CSP), formal models of computational dependencies, models of shared memory consistency
Distributed systems: network- and node-based faults, availability, distributed system and service design tradeoffs, examples of distributed algorithms (election, discovery)

P.S. Pacheco. An Introduction to Parallel Programming, Morgan Kaufman, 2011.
M. J. Quinn. Parallel Programing in C with MPI and OpenMP. Mc Graw Hill, 2003.
B.R. Gaster et. al. Heterogeneous computing with OpenCL. Morgan Kaufmann, 2013.
G. Couloris et al. Distributed Systems: Concepts and Design. Pearson, 2012.

To get the basic theoretical and practival knowledge from the areas of parallel and distributed systems, parallel programming and processing. To understand computer networks, inter-process communication and features of parallel algorithm design. To learn programming with pThreads, OpenMP, CUDA, and MPI. To understand Grid and concept of cloud computing. To realize future trends.

Knowledge and understanding:
The course focuses on the basic concepts of parallel and distributed systems, their architecture, communication, and coordination among elements. Important part of the course is in theorethical analysis and programming of important parallel agorithms, including parallel decomposition, theoretical analysis based on formal theoretical models, and perofrmance evaluation.
Application:
Student with skills gained in this course will be capable of designing, programming, and evaluating parallel and or distributed systems.

Reflection:
Awareness and understanding of connection between the theory and its application on parallel and distributed systems.

Transferable skills:
Capability of reading and understanding domestic and foreign technical literature, gathering and interpreting data, identifying and solving problems, critical analysis and evaluation of the usefulness of parallelization of distributed system and/or algorithms.

Lectures, laboratories, homework

one project
Two examinations during semester or oral examination active participation on lectures
grading: 5 (fail), 6-10 (pass) (according to the Statute of UL)

LOTRIČ, Uroš, DOBNIKAR, Andrej. Parallel implementations of recurrent neural network learning. V: KOLEHMAINEN, Mikko (ur.), TOIVANEN, Pekka (ur.), BELICZYŃSKI, Bartĺomiej (ur.). Adaptive and natural computing algorithms : 9th international conference, ICANNGA 2009, Kuopio, Finland, April 23-25, 2009 : revised selected papers, (Lecture notes in computer science, ISSN 0302-9743, 5495). Berlin, Heidelberg, New York: Springer, cop. 2009, str. 99-108, ilustr. [COBISS-SI-ID 7346004]
SILVA, Catarina, LOTRIČ, Uroš, RIBEIRO, Bernardete, DOBNIKAR, Andrej. Distributed text classification with an ensemble kernel-based learning approach. IEEE transactions on systems, man and cybernetics. Pt. C, Applications and reviews, ISSN 1094-6977. [Print ed.], May 2010, vol. 40, no. 3, str. 287-297. [COBISS-SI-ID 7683668]
LOTRIČ, Uroš, BULIĆ, Patricio. Applicability of approximate multipliers in hardware neural networks. Neurocomputing, ISSN 0925-2312. [Print ed.], Nov. 2012, vol. 96, str. 57-65, ilustr. [COBISS-SI-ID 9160276]
SLUGA, Davor, CURK, Tomaž, ZUPAN, Blaž, LOTRIČ, Uroš. Acceleration of information-theoretic data analysis with graphics processing units. Przeglęad Elektrotechniczny, ISSN 0033-2097, 2012, no. 2, str. 136-139, ilustr. [COBISS-SI-ID 8952148]
CANKAR, Matija, ARTAČ, Matej, ŠTERK, Marjan, LOTRIČ, Uroš, SLIVNIK, Boštjan. Co-allocation with collective requests in grid systems. Journal for universal computer science, ISSN 0948-6968, 2013, vol. 19, no. 3, str. 282-300, ilustr. [COBISS-SI-ID 9797972]
Uroš Lotrič: